[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:57:42 dist-test-slave-v3hj krb5kdc[24088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:57:42 dist-test-slave-v3hj krb5kdc[24088](info): set up 2 sockets
Mar 20 02:57:42 dist-test-slave-v3hj krb5kdc[24088](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260320 02:57:44.118664 24081 mini_kdc.cc:121] Time spent starting KDC: real 1.898s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:57:44 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:57:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:44.274374 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:40899
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:44.416466 24104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:44.416728 24104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:44.416806 24104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:44.420280 24104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:57:44.420352 24104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:44.420383 24104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:44.420408 24104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:57:44.420439 24104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:57:44.426075 24104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:40899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:40899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:44.427316 24104 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:44.428279 24104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:44.435958 24109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:44.436071 24112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:44.436295 24110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:44.436286 24104 server_base.cc:1061] running on GCE node
I20260320 02:57:44.436800 24104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:44.437834 24104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:44.439116 24104 hybrid_clock.cc:648] HybridClock initialized: now 1773975464439087 us; error 40 us; skew 500 ppm
Mar 20 02:57:44 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975464, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:44.442322 24104 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:57:44.443508 24104 webserver.cc:492] Webserver started at http://127.23.132.126:42207/ using document root <none> and password file <none>
I20260320 02:57:44.444067 24104 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:44.444116 24104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:44.444357 24104 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:44.446166 24104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "666e8e719aff478eb36470ef7084e54e"
format_stamp: "Formatted at 2026-03-20 02:57:44 on dist-test-slave-v3hj"
server_key: "b05f448beb17eab32f3600d28e3bfcf4"
server_key_iv: "cd7d7c647778cd9e592b059bf9482b25"
server_key_version: "encryptionkey@0"
I20260320 02:57:44.446658 24104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "666e8e719aff478eb36470ef7084e54e"
format_stamp: "Formatted at 2026-03-20 02:57:44 on dist-test-slave-v3hj"
server_key: "b05f448beb17eab32f3600d28e3bfcf4"
server_key_iv: "cd7d7c647778cd9e592b059bf9482b25"
server_key_version: "encryptionkey@0"
I20260320 02:57:44.450332 24104 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:44.453521 24119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:44.455333 24104 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:44.455597 24104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "666e8e719aff478eb36470ef7084e54e"
format_stamp: "Formatted at 2026-03-20 02:57:44 on dist-test-slave-v3hj"
server_key: "b05f448beb17eab32f3600d28e3bfcf4"
server_key_iv: "cd7d7c647778cd9e592b059bf9482b25"
server_key_version: "encryptionkey@0"
I20260320 02:57:44.455737 24104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:44.477540 24104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:44.480975 24104 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:44.481194 24104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:44.491228 24104 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:40899
I20260320 02:57:44.491258 24171 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:40899 every 8 connection(s)
I20260320 02:57:44.492379 24104 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:57:44.493989 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24104
I20260320 02:57:44.494122 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:57:44.495334 24081 external_mini_cluster.cc:1468] Setting key 9a756ea1c13dc099051c2af8a411d6de
I20260320 02:57:44.495761 24172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:44.502566 24172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e: Bootstrap starting.
I20260320 02:57:44.505287 24172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:44.506210 24172 log.cc:826] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:44.509433 24172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e: No bootstrap required, opened a new log
Mar 20 02:57:44 dist-test-slave-v3hj krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:44.512159 24172 raft_consensus.cc:359] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e8e719aff478eb36470ef7084e54e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 40899 } }
I20260320 02:57:44.512352 24172 raft_consensus.cc:385] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:44.512403 24172 raft_consensus.cc:740] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666e8e719aff478eb36470ef7084e54e, State: Initialized, Role: FOLLOWER
I20260320 02:57:44.512821 24172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e8e719aff478eb36470ef7084e54e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 40899 } }
I20260320 02:57:44.512950 24172 raft_consensus.cc:399] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:57:44.513046 24172 raft_consensus.cc:493] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:57:44.513161 24172 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:44.514038 24172 raft_consensus.cc:515] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e8e719aff478eb36470ef7084e54e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 40899 } }
I20260320 02:57:44.514375 24172 leader_election.cc:304] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666e8e719aff478eb36470ef7084e54e; no voters:
I20260320 02:57:44.514688 24172 leader_election.cc:290] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:57:44.514788 24177 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:44.514988 24177 raft_consensus.cc:697] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [term 1 LEADER]: Becoming Leader. State: Replica: 666e8e719aff478eb36470ef7084e54e, State: Running, Role: LEADER
I20260320 02:57:44.515326 24177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e8e719aff478eb36470ef7084e54e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 40899 } }
I20260320 02:57:44.515925 24172 sys_catalog.cc:565] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:57:44.517535 24178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666e8e719aff478eb36470ef7084e54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e8e719aff478eb36470ef7084e54e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 40899 } } }
I20260320 02:57:44.517659 24178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:44.518469 24179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666e8e719aff478eb36470ef7084e54e. Latest consensus state: current_term: 1 leader_uuid: "666e8e719aff478eb36470ef7084e54e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666e8e719aff478eb36470ef7084e54e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 40899 } } }
I20260320 02:57:44.518558 24179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:44.519333 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:44.497943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.23.132.126:40899)
0320 02:57:44.498429 (+ 486us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:44.498440 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:44.498823 (+ 383us) server_negotiation.cc:408] Connection header received
0320 02:57:44.501563 (+ 2740us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:44.501577 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:44.501898 (+ 321us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:44.502146 (+ 248us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:44.504420 (+ 2274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:44.505225 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:44.506238 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:44.506634 (+ 396us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:44.512086 (+ 5452us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:44.512109 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:44.512120 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:44.512160 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:44.514361 (+ 2201us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:44.515360 (+ 999us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:44.515372 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:44.515377 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:44.515492 (+ 115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:44.516310 (+ 818us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:44.516313 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:44.516314 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:44.516799 (+ 485us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:44.517045 (+ 246us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:44.518313 (+ 1268us) server_negotiation.cc:300] Negotiation successful
0320 02:57:44.518576 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":123,"threads_started":1}
W20260320 02:57:44.521350 24193 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:57:44.521420 24193 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:57:44.521474 24186 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:57:44.522778 24186 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:57:44.529687 24186 catalog_manager.cc:1357] Generated new cluster ID: c36bec25e7db46acb6056ec2c543067a
I20260320 02:57:44.529887 24186 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:57:44.548902 24186 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:57:44.550227 24186 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:57:44.561192 24186 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 666e8e719aff478eb36470ef7084e54e: Generated new TSK 0
I20260320 02:57:44.562170 24186 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:44.641985 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:40899
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:44.759420 24201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:44.759658 24201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:44.759719 24201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:44.763813 24201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:44.763948 24201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:44.764043 24201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:57:44.768731 24201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:40899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:44.769930 24201 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:44.770799 24201 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:44.777602 24207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:44.777808 24206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:44.777815 24209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:44.777896 24201 server_base.cc:1061] running on GCE node
I20260320 02:57:44.778923 24201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:44.779733 24201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:44.780982 24201 hybrid_clock.cc:648] HybridClock initialized: now 1773975464780956 us; error 54 us; skew 500 ppm
Mar 20 02:57:44 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975464, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:44.784507 24201 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:57:44.786155 24201 webserver.cc:492] Webserver started at http://127.23.132.65:39847/ using document root <none> and password file <none>
I20260320 02:57:44.786772 24201 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:44.786828 24201 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:44.787016 24201 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:44.788823 24201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "e5e70c5905d04f5da80962d005f33054"
format_stamp: "Formatted at 2026-03-20 02:57:44 on dist-test-slave-v3hj"
server_key: "12527d3c3a713adc6bdbf74a8de046d5"
server_key_iv: "1991a2a1d3f7d4f7386589054ae6b6ca"
server_key_version: "encryptionkey@0"
I20260320 02:57:44.789291 24201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e70c5905d04f5da80962d005f33054"
format_stamp: "Formatted at 2026-03-20 02:57:44 on dist-test-slave-v3hj"
server_key: "12527d3c3a713adc6bdbf74a8de046d5"
server_key_iv: "1991a2a1d3f7d4f7386589054ae6b6ca"
server_key_version: "encryptionkey@0"
I20260320 02:57:44.793386 24201 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260320 02:57:44.796306 24217 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:44.797574 24201 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:57:44.797683 24201 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "e5e70c5905d04f5da80962d005f33054"
format_stamp: "Formatted at 2026-03-20 02:57:44 on dist-test-slave-v3hj"
server_key: "12527d3c3a713adc6bdbf74a8de046d5"
server_key_iv: "1991a2a1d3f7d4f7386589054ae6b6ca"
server_key_version: "encryptionkey@0"
I20260320 02:57:44.797866 24201 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:44.819242 24201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:44.823006 24201 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:44.823236 24201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:44.824306 24201 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:57:44.826169 24201 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:44.826243 24201 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:44.826294 24201 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:44.826308 24201 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:44.840401 24201 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:42431
I20260320 02:57:44.840420 24331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:42431 every 8 connection(s)
I20260320 02:57:44.842187 24201 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:57:44.849625 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24201
I20260320 02:57:44.849737 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:57:44.850023 24081 external_mini_cluster.cc:1468] Setting key 38785716105b10f641f1dd60a7ca6cff
Mar 20 02:57:44 dist-test-slave-v3hj krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975464, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:44.856326 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:44.844157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:42743 (local address 127.23.132.126:40899)
0320 02:57:44.844292 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:44.844295 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:44.845040 (+ 745us) server_negotiation.cc:408] Connection header received
0320 02:57:44.845844 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:44.845848 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:44.845907 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:44.845979 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:44.847602 (+ 1623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:44.848110 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:44.848740 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:44.848918 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:44.851893 (+ 2975us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:44.851915 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:44.851918 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:44.851946 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:44.853607 (+ 1661us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:44.854291 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:44.854295 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:44.854298 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:44.854357 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:44.854893 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:44.854897 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:44.854899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:44.855146 (+ 247us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:44.855245 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:44.856001 (+ 756us) server_negotiation.cc:300] Negotiation successful
0320 02:57:44.856154 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260320 02:57:44.857105 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:44.844347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:40899 (local address 127.23.132.65:42743)
0320 02:57:44.844849 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:44.844881 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:44.845631 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:44.846141 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:44.846150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:44.846521 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:44.847453 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:44.847466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:44.848256 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:44.848260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:44.848628 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:44.848635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:44.848828 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:44.849393 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:44.849426 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:44.851639 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:44.853860 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:44.853869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:44.853885 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:44.854177 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:44.854530 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:44.854533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:44.854535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:44.854647 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:44.855267 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:44.855272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:44.855517 (+ 245us) client_negotiation.cc:769] Sending connection context
0320 02:57:44.855929 (+ 412us) client_negotiation.cc:240] Negotiation successful
0320 02:57:44.856255 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":106,"threads_started":1}
I20260320 02:57:44.858314 24332 heartbeater.cc:344] Connected to a master server at 127.23.132.126:40899
I20260320 02:57:44.858841 24332 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:57:44.859393 24332 heartbeater.cc:507] Master 127.23.132.126:40899 requested a full tablet report, sending...
I20260320 02:57:44.860958 24136 ts_manager.cc:194] Registered new tserver with Master: e5e70c5905d04f5da80962d005f33054 (127.23.132.65:42431)
I20260320 02:57:44.862329 24136 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:42743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:44.912930 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:40899
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:45.023389 24339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:45.023618 24339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:45.023705 24339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:45.027182 24339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:45.027254 24339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:45.027359 24339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:57:45.032522 24339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:40899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:45.033977 24339 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:45.035000 24339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:45.042965 24347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:45.042969 24344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:45.042965 24345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:45.043601 24339 server_base.cc:1061] running on GCE node
I20260320 02:57:45.045185 24339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:45.045887 24339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:45.047268 24339 hybrid_clock.cc:648] HybridClock initialized: now 1773975465047167 us; error 109 us; skew 500 ppm
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:45.052191 24339 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:57:45.053462 24339 webserver.cc:492] Webserver started at http://127.23.132.66:36101/ using document root <none> and password file <none>
I20260320 02:57:45.054100 24339 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:45.054157 24339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:45.054344 24339 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:45.056094 24339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "952c0e37de354bfe8dce216e08e9a237"
format_stamp: "Formatted at 2026-03-20 02:57:45 on dist-test-slave-v3hj"
server_key: "ec1acbbc01e6f0b055b1f50fef446c39"
server_key_iv: "0528a44492041b9e39d681aba9ed117d"
server_key_version: "encryptionkey@0"
I20260320 02:57:45.056571 24339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "952c0e37de354bfe8dce216e08e9a237"
format_stamp: "Formatted at 2026-03-20 02:57:45 on dist-test-slave-v3hj"
server_key: "ec1acbbc01e6f0b055b1f50fef446c39"
server_key_iv: "0528a44492041b9e39d681aba9ed117d"
server_key_version: "encryptionkey@0"
I20260320 02:57:45.060659 24339 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:57:45.064548 24354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:45.066208 24339 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:57:45.066336 24339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "952c0e37de354bfe8dce216e08e9a237"
format_stamp: "Formatted at 2026-03-20 02:57:45 on dist-test-slave-v3hj"
server_key: "ec1acbbc01e6f0b055b1f50fef446c39"
server_key_iv: "0528a44492041b9e39d681aba9ed117d"
server_key_version: "encryptionkey@0"
I20260320 02:57:45.066462 24339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:45.079998 24339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:45.083585 24339 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:45.083793 24339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:45.084852 24339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:57:45.086508 24339 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:45.086673 24339 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:45.086795 24339 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:45.086825 24339 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:45.099269 24339 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:46435
I20260320 02:57:45.099367 24468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:46435 every 8 connection(s)
I20260320 02:57:45.100363 24339 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:45.110486 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24339
I20260320 02:57:45.110595 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:57:45.111465 24081 external_mini_cluster.cc:1468] Setting key c630e1962bccda9a7f9bdf25c56e4613
I20260320 02:57:45.115012 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:45.102507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:48201 (local address 127.23.132.126:40899)
0320 02:57:45.102697 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:45.102701 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:45.103298 (+ 597us) server_negotiation.cc:408] Connection header received
0320 02:57:45.104218 (+ 920us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:45.104221 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:45.104277 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:45.104355 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:45.106170 (+ 1815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.106693 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:45.107389 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.107531 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:45.110314 (+ 2783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:45.110340 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:45.110343 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:45.110388 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:45.112612 (+ 2224us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.113200 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.113204 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.113206 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.113341 (+ 135us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.113839 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.113843 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.113845 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.114030 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:45.114125 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:45.114655 (+ 530us) server_negotiation.cc:300] Negotiation successful
0320 02:57:45.114848 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260320 02:57:45.115702 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:45.102496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:40899 (local address 127.23.132.66:48201)
0320 02:57:45.103097 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:45.103143 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:45.103992 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:45.104521 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:45.104529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:45.104992 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:45.106000 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:45.106015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.106844 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:45.106847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:45.107256 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:45.107264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.107430 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:45.107987 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:45.108011 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:45.110057 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:45.112793 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:45.112801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:45.112814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:45.113088 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:45.113581 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:45.113585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:45.113587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:45.113714 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:45.114148 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:45.114155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:45.114403 (+ 248us) client_negotiation.cc:769] Sending connection context
0320 02:57:45.114601 (+ 198us) client_negotiation.cc:240] Negotiation successful
0320 02:57:45.114892 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260320 02:57:45.117271 24469 heartbeater.cc:344] Connected to a master server at 127.23.132.126:40899
I20260320 02:57:45.117549 24469 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:45.118705 24469 heartbeater.cc:507] Master 127.23.132.126:40899 requested a full tablet report, sending...
I20260320 02:57:45.120081 24136 ts_manager.cc:194] Registered new tserver with Master: 952c0e37de354bfe8dce216e08e9a237 (127.23.132.66:46435)
I20260320 02:57:45.121101 24136 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:48201
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:45.183012 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:40899
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:45.313666 24476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:45.313957 24476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:45.314021 24476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:45.318797 24476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:45.318878 24476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:45.319000 24476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:57:45.327575 24476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:40899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:45.330195 24476 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:45.332055 24476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:45.343043 24484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:45.343043 24482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:45.343449 24481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:45.343931 24476 server_base.cc:1061] running on GCE node
I20260320 02:57:45.344362 24476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:45.345010 24476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:45.346395 24476 hybrid_clock.cc:648] HybridClock initialized: now 1773975465346360 us; error 49 us; skew 500 ppm
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:45.350251 24476 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:57:45.351816 24476 webserver.cc:492] Webserver started at http://127.23.132.67:34957/ using document root <none> and password file <none>
I20260320 02:57:45.352393 24476 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:45.352447 24476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:45.352663 24476 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:45.354688 24476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "a219dd65cfa74450aa8727d74469b22e"
format_stamp: "Formatted at 2026-03-20 02:57:45 on dist-test-slave-v3hj"
server_key: "f3ba3449174f11b3455ef975aa65ac01"
server_key_iv: "84b0aebb78bbd5fde759bd7c04cb6561"
server_key_version: "encryptionkey@0"
I20260320 02:57:45.355161 24476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "a219dd65cfa74450aa8727d74469b22e"
format_stamp: "Formatted at 2026-03-20 02:57:45 on dist-test-slave-v3hj"
server_key: "f3ba3449174f11b3455ef975aa65ac01"
server_key_iv: "84b0aebb78bbd5fde759bd7c04cb6561"
server_key_version: "encryptionkey@0"
I20260320 02:57:45.360415 24476 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260320 02:57:45.365593 24491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:45.367054 24476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:57:45.367172 24476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "a219dd65cfa74450aa8727d74469b22e"
format_stamp: "Formatted at 2026-03-20 02:57:45 on dist-test-slave-v3hj"
server_key: "f3ba3449174f11b3455ef975aa65ac01"
server_key_iv: "84b0aebb78bbd5fde759bd7c04cb6561"
server_key_version: "encryptionkey@0"
I20260320 02:57:45.367262 24476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:45.394345 24476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:45.398073 24476 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:45.398260 24476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:45.398805 24476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:57:45.399691 24476 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:45.399744 24476 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:45.399782 24476 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:45.399796 24476 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:45.412843 24476 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:33087
I20260320 02:57:45.412916 24605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:33087 every 8 connection(s)
I20260320 02:57:45.414235 24476 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:57:45.422094 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24476
I20260320 02:57:45.422202 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:57:45.422425 24081 external_mini_cluster.cc:1468] Setting key d9901e633d653b996f74d35f804f862b
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:45.429147 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:45.416477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:52929 (local address 127.23.132.126:40899)
0320 02:57:45.416703 (+ 226us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:45.416707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:45.417012 (+ 305us) server_negotiation.cc:408] Connection header received
0320 02:57:45.417998 (+ 986us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:45.418002 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:45.418054 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:45.418175 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:45.420057 (+ 1882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.420560 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:45.421442 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.421856 (+ 414us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:45.424067 (+ 2211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:45.424092 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:45.424096 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:45.424127 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:45.425916 (+ 1789us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.426457 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.426460 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.426461 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.426503 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.426990 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.426997 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.426999 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.427388 (+ 389us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:45.427471 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:45.428645 (+ 1174us) server_negotiation.cc:300] Negotiation successful
0320 02:57:45.428934 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260320 02:57:45.430030 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:45.416267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:40899 (local address 127.23.132.67:52929)
0320 02:57:45.416732 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:45.416773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:45.417587 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:45.418404 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:45.418412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:45.418895 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:45.419870 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:45.419888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.420752 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:45.420755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:45.421153 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:45.421160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.421330 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:45.422050 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:45.422074 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:45.423784 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:45.426068 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:45.426074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:45.426087 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:45.426378 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:45.426591 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:45.426594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:45.426595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:45.426783 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:45.427593 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:45.427602 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:45.428132 (+ 530us) client_negotiation.cc:769] Sending connection context
0320 02:57:45.428536 (+ 404us) client_negotiation.cc:240] Negotiation successful
0320 02:57:45.428899 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260320 02:57:45.432093 24606 heartbeater.cc:344] Connected to a master server at 127.23.132.126:40899
I20260320 02:57:45.432382 24606 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:45.432993 24606 heartbeater.cc:507] Master 127.23.132.126:40899 requested a full tablet report, sending...
I20260320 02:57:45.434614 24136 ts_manager.cc:194] Registered new tserver with Master: a219dd65cfa74450aa8727d74469b22e (127.23.132.67:33087)
I20260320 02:57:45.435227 24136 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:52929
I20260320 02:57:45.439735 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:57:45.477521 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:45.460015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.23.132.65:42431)
0320 02:57:45.460639 (+ 624us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:45.460650 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:45.460927 (+ 277us) server_negotiation.cc:408] Connection header received
0320 02:57:45.461049 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:45.461055 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:45.462786 (+ 1731us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:45.463098 (+ 312us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:45.465017 (+ 1919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.467461 (+ 2444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:45.468462 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.469013 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:45.472612 (+ 3599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:45.472677 (+ 65us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:45.472690 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:45.472722 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:45.474599 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.475171 (+ 572us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.475177 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.475178 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.475234 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.475789 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.475792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.475794 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.476011 (+ 217us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:45.476175 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:45.476660 (+ 485us) server_negotiation.cc:300] Negotiation successful
0320 02:57:45.476899 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":145,"threads_started":1}
W20260320 02:57:45.478621 24245 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44382
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:57:45.512247 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:45.502475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.23.132.65:42431)
0320 02:57:45.502639 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:45.502643 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:45.502793 (+ 150us) server_negotiation.cc:408] Connection header received
0320 02:57:45.502861 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:45.502864 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:45.502916 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:45.502993 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:45.504075 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.504575 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:45.505586 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:45.505780 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:45.507972 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:45.507993 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:45.507995 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:45.508019 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:45.510412 (+ 2393us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.510842 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.510846 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.510847 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.510891 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:45.511123 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:45.511125 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:45.511126 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:45.511258 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:45.511355 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:45.511946 (+ 591us) server_negotiation.cc:300] Negotiation successful
0320 02:57:45.512061 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260320 02:57:45.513985 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24201
I20260320 02:57:45.521770 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24339
I20260320 02:57:45.531704 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24476
I20260320 02:57:45.538733 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24104
2026-03-20T02:57:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3353 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24620](info): set up 2 sockets
Mar 20 02:57:45 dist-test-slave-v3hj krb5kdc[24620](info): commencing operation
krb5kdc: starting...
W20260320 02:57:47.498358 24081 mini_kdc.cc:121] Time spent starting KDC: real 1.931s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:57:47 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:57:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:47.678128 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:43669
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:47.818501 24636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:47.819144 24636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:47.819301 24636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:47.824172 24636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:57:47.824280 24636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:47.824304 24636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:47.824323 24636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:57:47.824339 24636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:57:47.830982 24636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:43669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:47.832461 24636 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:47.833536 24636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:47.840162 24642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:47.840144 24645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:47.840145 24643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:47.841456 24636 server_base.cc:1061] running on GCE node
I20260320 02:57:47.842382 24636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:47.843408 24636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:47.844623 24636 hybrid_clock.cc:648] HybridClock initialized: now 1773975467844592 us; error 57 us; skew 500 ppm
Mar 20 02:57:47 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975467, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:47.848789 24636 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:57:47.851749 24636 webserver.cc:492] Webserver started at http://127.23.132.126:40431/ using document root <none> and password file <none>
I20260320 02:57:47.852845 24636 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:47.852952 24636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:47.853515 24636 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:47.857160 24636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "6549b042b1944cb6b158cfd026a29f64"
format_stamp: "Formatted at 2026-03-20 02:57:47 on dist-test-slave-v3hj"
server_key: "197c90a20c6ebd80357f3c9fddc0bc80"
server_key_iv: "c0e96970cf98ade5c927102dba997338"
server_key_version: "encryptionkey@0"
I20260320 02:57:47.858204 24636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "6549b042b1944cb6b158cfd026a29f64"
format_stamp: "Formatted at 2026-03-20 02:57:47 on dist-test-slave-v3hj"
server_key: "197c90a20c6ebd80357f3c9fddc0bc80"
server_key_iv: "c0e96970cf98ade5c927102dba997338"
server_key_version: "encryptionkey@0"
I20260320 02:57:47.862911 24636 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:57:47.867103 24652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:47.869479 24636 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:57:47.869637 24636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "6549b042b1944cb6b158cfd026a29f64"
format_stamp: "Formatted at 2026-03-20 02:57:47 on dist-test-slave-v3hj"
server_key: "197c90a20c6ebd80357f3c9fddc0bc80"
server_key_iv: "c0e96970cf98ade5c927102dba997338"
server_key_version: "encryptionkey@0"
I20260320 02:57:47.869755 24636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:47.880234 24636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:47.889665 24636 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:47.889955 24636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:47.899412 24636 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:43669
I20260320 02:57:47.899420 24704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:43669 every 8 connection(s)
I20260320 02:57:47.900800 24636 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:57:47.904839 24705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:47.907270 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24636
I20260320 02:57:47.907364 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:57:47.907574 24081 external_mini_cluster.cc:1468] Setting key 3356ba88264497aa1f5516b5f7ea96aa
I20260320 02:57:47.912407 24705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64: Bootstrap starting.
I20260320 02:57:47.915306 24705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:47.916355 24705 log.cc:826] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64: Log is configured to *not* fsync() on all Append() calls
Mar 20 02:57:47 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:47.919108 24705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64: No bootstrap required, opened a new log
I20260320 02:57:47.922979 24705 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6549b042b1944cb6b158cfd026a29f64" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43669 } }
I20260320 02:57:47.923214 24705 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:47.923265 24705 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6549b042b1944cb6b158cfd026a29f64, State: Initialized, Role: FOLLOWER
I20260320 02:57:47.924182 24705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6549b042b1944cb6b158cfd026a29f64" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43669 } }
I20260320 02:57:47.924368 24705 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:57:47.924525 24705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:57:47.924697 24705 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:47.925200 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:47.909588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.23.132.126:43669)
0320 02:57:47.910129 (+ 541us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:47.910139 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:47.910178 (+ 39us) server_negotiation.cc:408] Connection header received
0320 02:57:47.910746 (+ 568us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:47.910761 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:47.911063 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:47.911349 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:47.912309 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:47.913472 (+ 1163us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:47.915647 (+ 2175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:47.916025 (+ 378us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:47.918920 (+ 2895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:47.919029 (+ 109us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:47.919115 (+ 86us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:47.919161 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:47.921770 (+ 2609us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:47.922255 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:47.922261 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:47.922266 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:47.922344 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:47.922569 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:47.922572 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:47.922574 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:47.922949 (+ 375us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:47.923198 (+ 249us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:47.923839 (+ 641us) server_negotiation.cc:300] Negotiation successful
0320 02:57:47.924272 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":190,"threads_started":1}
I20260320 02:57:47.925822 24705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6549b042b1944cb6b158cfd026a29f64" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43669 } }
I20260320 02:57:47.926218 24705 leader_election.cc:304] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6549b042b1944cb6b158cfd026a29f64; no voters:
I20260320 02:57:47.927001 24705 leader_election.cc:290] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:57:47.927121 24710 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:47.927477 24710 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [term 1 LEADER]: Becoming Leader. State: Replica: 6549b042b1944cb6b158cfd026a29f64, State: Running, Role: LEADER
I20260320 02:57:47.928225 24710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6549b042b1944cb6b158cfd026a29f64" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43669 } }
I20260320 02:57:47.929298 24705 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:57:47.932516 24711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6549b042b1944cb6b158cfd026a29f64. Latest consensus state: current_term: 1 leader_uuid: "6549b042b1944cb6b158cfd026a29f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6549b042b1944cb6b158cfd026a29f64" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43669 } } }
I20260320 02:57:47.932724 24711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:47.933492 24712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6549b042b1944cb6b158cfd026a29f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6549b042b1944cb6b158cfd026a29f64" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43669 } } }
I20260320 02:57:47.933596 24712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64 [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:47.933960 24719 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:57:47.938652 24719 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:57:47.947844 24719 catalog_manager.cc:1357] Generated new cluster ID: 98c6de2a7dd04cd8a0723fbebd50a353
I20260320 02:57:47.947947 24719 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:57:47.973387 24719 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:57:47.975214 24719 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:57:47.984637 24719 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6549b042b1944cb6b158cfd026a29f64: Generated new TSK 0
I20260320 02:57:47.985666 24719 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:48.079567 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:43669
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:48.212136 24733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:48.212399 24733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:48.212459 24733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:48.216487 24733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:48.216746 24733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:48.217398 24733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:57:48.223922 24733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:43669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:48.225111 24733 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:48.226260 24733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:48.240300 24739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:48.240300 24741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:48.240746 24738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:48.242413 24733 server_base.cc:1061] running on GCE node
I20260320 02:57:48.242911 24733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:48.243532 24733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:48.247115 24733 hybrid_clock.cc:648] HybridClock initialized: now 1773975468246876 us; error 246 us; skew 500 ppm
Mar 20 02:57:48 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:48.251312 24733 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:57:48.254235 24733 webserver.cc:492] Webserver started at http://127.23.132.65:34101/ using document root <none> and password file <none>
I20260320 02:57:48.255360 24733 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:48.255435 24733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:48.255651 24733 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:48.257510 24733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "ba30f437e481fa8e1aa89eaf6b50b985"
server_key_iv: "10d360b6c99b80f32cf6d8842700010a"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.258055 24733 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "ba30f437e481fa8e1aa89eaf6b50b985"
server_key_iv: "10d360b6c99b80f32cf6d8842700010a"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.262277 24733 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260320 02:57:48.264922 24748 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.267153 24733 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:48.267292 24733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "ba30f437e481fa8e1aa89eaf6b50b985"
server_key_iv: "10d360b6c99b80f32cf6d8842700010a"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.267405 24733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:48.292464 24733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:48.297524 24733 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:48.298401 24733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:48.299283 24733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:57:48.302127 24733 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:48.302193 24733 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.302237 24733 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:48.302250 24733 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.318522 24733 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:43723
I20260320 02:57:48.318536 24862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:43723 every 8 connection(s)
I20260320 02:57:48.319890 24733 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:57:48.330123 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24733
I20260320 02:57:48.330228 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:57:48.330456 24081 external_mini_cluster.cc:1468] Setting key 901ade1dceabd0a43082b485417a93af
Mar 20 02:57:48 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:48.339603 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.322667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:60787 (local address 127.23.132.126:43669)
0320 02:57:48.322838 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.322843 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.323570 (+ 727us) server_negotiation.cc:408] Connection header received
0320 02:57:48.324557 (+ 987us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.324562 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.324647 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.324765 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:48.327400 (+ 2635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.327919 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.328677 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.328821 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.331973 (+ 3152us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:48.331988 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:48.331992 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:48.332020 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:48.336266 (+ 4246us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.337367 (+ 1101us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.337371 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.337372 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.337505 (+ 133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.338094 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.338097 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.338099 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.338493 (+ 394us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:48.338600 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.339274 (+ 674us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.339432 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260320 02:57:48.340247 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.322901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:43669 (local address 127.23.132.65:60787)
0320 02:57:48.323418 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:48.323455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:48.324262 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:48.324997 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:48.325005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:48.325542 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:48.327171 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.327191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.328129 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.328132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:48.328520 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.328527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.328700 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.329618 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:48.329644 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:48.331666 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:48.336604 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:48.336613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:48.336630 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:48.337151 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:48.337617 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:48.337621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:48.337623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:48.337836 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:48.338636 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:48.338643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:48.339007 (+ 364us) client_negotiation.cc:769] Sending connection context
0320 02:57:48.339217 (+ 210us) client_negotiation.cc:240] Negotiation successful
0320 02:57:48.339524 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":125,"threads_started":1}
I20260320 02:57:48.342633 24863 heartbeater.cc:344] Connected to a master server at 127.23.132.126:43669
I20260320 02:57:48.342883 24863 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:48.343451 24863 heartbeater.cc:507] Master 127.23.132.126:43669 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:57:48.345022 24669 ts_manager.cc:194] Registered new tserver with Master: 8db4d790d7ff4e0bb86032cf65c7c1fa (127.23.132.65:43723)
I20260320 02:57:48.346225 24669 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:60787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:48.418596 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:43669
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:48.539950 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:48.540205 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:48.540289 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:48.544535 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:48.544638 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:48.544767 24870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:57:48.551819 24870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:43669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:48.553187 24870 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:48.554131 24870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:48.562527 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:48.562527 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:48.562808 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:48.563095 24870 server_base.cc:1061] running on GCE node
I20260320 02:57:48.563460 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:48.564796 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:48.565972 24870 hybrid_clock.cc:648] HybridClock initialized: now 1773975468565959 us; error 29 us; skew 500 ppm
Mar 20 02:57:48 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:48.568969 24870 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:57:48.570298 24870 webserver.cc:492] Webserver started at http://127.23.132.66:41361/ using document root <none> and password file <none>
I20260320 02:57:48.570912 24870 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:48.570968 24870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:48.571153 24870 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:48.572929 24870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "1dad660241b94db09a81a025e653b877"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "71db62df4612aa416fd52b238beecec5"
server_key_iv: "890b3df581eccd5f7cff45bcd01c891d"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.573419 24870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "1dad660241b94db09a81a025e653b877"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "71db62df4612aa416fd52b238beecec5"
server_key_iv: "890b3df581eccd5f7cff45bcd01c891d"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.577014 24870 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260320 02:57:48.580098 24885 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.581585 24870 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260320 02:57:48.581703 24870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "1dad660241b94db09a81a025e653b877"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "71db62df4612aa416fd52b238beecec5"
server_key_iv: "890b3df581eccd5f7cff45bcd01c891d"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.581879 24870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:48.612210 24870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:48.616981 24870 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:48.617187 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:48.617871 24870 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:57:48.618834 24870 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:48.618887 24870 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.618932 24870 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:48.618948 24870 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.633256 24870 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:39991
I20260320 02:57:48.633643 24999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:39991 every 8 connection(s)
I20260320 02:57:48.634359 24870 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:57:48.636415 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 24870
I20260320 02:57:48.636531 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:57:48.636775 24081 external_mini_cluster.cc:1468] Setting key 5bf148f56c38806b45ff0109a1c4e4ef
Mar 20 02:57:48 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:57:48.652415 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.637057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:50799 (local address 127.23.132.126:43669)
0320 02:57:48.637523 (+ 466us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.637529 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.638138 (+ 609us) server_negotiation.cc:408] Connection header received
0320 02:57:48.639724 (+ 1586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.639728 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.639777 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.639851 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:48.641987 (+ 2136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.642785 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.643592 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.643941 (+ 349us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.646802 (+ 2861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:48.646839 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:48.646843 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:48.646886 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:48.649350 (+ 2464us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.650364 (+ 1014us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.650369 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.650372 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.650445 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.650853 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.650856 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.650858 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.651224 (+ 366us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:48.651332 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.651988 (+ 656us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.652157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260320 02:57:48.652989 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.637154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:43669 (local address 127.23.132.66:50799)
0320 02:57:48.637916 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:48.637960 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:48.639458 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:48.640055 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:48.640068 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:48.640596 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:48.641826 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.641846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.643072 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.643076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:48.643467 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.643474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.643661 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.644230 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:48.644253 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:48.646509 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:48.649712 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:48.649722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:48.649739 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:48.650210 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:48.650603 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:48.650607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:48.650608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:48.650734 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:48.651365 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:48.651371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:48.651630 (+ 259us) client_negotiation.cc:769] Sending connection context
0320 02:57:48.651821 (+ 191us) client_negotiation.cc:240] Negotiation successful
0320 02:57:48.652061 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":263,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
I20260320 02:57:48.655068 25000 heartbeater.cc:344] Connected to a master server at 127.23.132.126:43669
I20260320 02:57:48.655633 25000 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:48.656386 25000 heartbeater.cc:507] Master 127.23.132.126:43669 requested a full tablet report, sending...
I20260320 02:57:48.658344 24669 ts_manager.cc:194] Registered new tserver with Master: 1dad660241b94db09a81a025e653b877 (127.23.132.66:39991)
I20260320 02:57:48.659116 24669 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:50799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:48.714699 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:43669
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:48.841130 25007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:48.841357 25007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:48.841423 25007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:48.846395 25007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:48.846491 25007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:48.846702 25007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:57:48.852334 25007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:43669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:48.853559 25007 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:48.854485 25007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:48.862409 25012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:48.862411 25013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:48.863236 25015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:48.864218 25007 server_base.cc:1061] running on GCE node
I20260320 02:57:48.864632 25007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:48.865226 25007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:48.866499 25007 hybrid_clock.cc:648] HybridClock initialized: now 1773975468866465 us; error 46 us; skew 500 ppm
Mar 20 02:57:48 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:48.870191 25007 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:57:48.871310 25007 webserver.cc:492] Webserver started at http://127.23.132.67:33267/ using document root <none> and password file <none>
I20260320 02:57:48.872265 25007 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:48.872330 25007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:48.872529 25007 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:48.874451 25007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "4d89b049c787452b81021e53564af253"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "be703f4fc5b7d6f7c008748dd9d3f41d"
server_key_iv: "23175f17a431409eef1a1c2bc935c88f"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.875334 25007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "4d89b049c787452b81021e53564af253"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "be703f4fc5b7d6f7c008748dd9d3f41d"
server_key_iv: "23175f17a431409eef1a1c2bc935c88f"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.879814 25007 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:57:48.883914 25022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.885215 25007 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:57:48.885330 25007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "4d89b049c787452b81021e53564af253"
format_stamp: "Formatted at 2026-03-20 02:57:48 on dist-test-slave-v3hj"
server_key: "be703f4fc5b7d6f7c008748dd9d3f41d"
server_key_iv: "23175f17a431409eef1a1c2bc935c88f"
server_key_version: "encryptionkey@0"
I20260320 02:57:48.885452 25007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:48.897015 25007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:48.901103 25007 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:48.901300 25007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:48.902014 25007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:57:48.903213 25007 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:48.903275 25007 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.903323 25007 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:48.903342 25007 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:48.915529 25007 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:36771
I20260320 02:57:48.915627 25135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:36771 every 8 connection(s)
I20260320 02:57:48.916525 25007 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:57:48.923111 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25007
I20260320 02:57:48.923249 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:57:48.923498 24081 external_mini_cluster.cc:1468] Setting key 945a1565ef9dfcddea225ea7f3f9de37
Mar 20 02:57:48 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:48.934981 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.919863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:41373 (local address 127.23.132.126:43669)
0320 02:57:48.920072 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.920075 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.920744 (+ 669us) server_negotiation.cc:408] Connection header received
0320 02:57:48.921661 (+ 917us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.921665 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.921727 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.921843 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:48.923984 (+ 2141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.924520 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.926271 (+ 1751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.926457 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.929189 (+ 2732us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:48.929208 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:48.929211 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:48.929239 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:48.931184 (+ 1945us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.932242 (+ 1058us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.932251 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.932252 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.932326 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.933284 (+ 958us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.933291 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.933293 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.933507 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:48.933918 (+ 411us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.934653 (+ 735us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.934808 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260320 02:57:48.936110 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.919994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:43669 (local address 127.23.132.67:41373)
0320 02:57:48.920505 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:48.920542 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:48.921341 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:48.922143 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:48.922151 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:48.922552 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:48.923726 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.923745 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.924714 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.924719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:48.925991 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.926007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.926362 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.927058 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:48.927348 (+ 290us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:48.929011 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:48.931690 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:48.931700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:48.931711 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:48.931991 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:48.932600 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:48.932610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:48.932612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:48.932961 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:48.933992 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:48.933998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:48.934364 (+ 366us) client_negotiation.cc:769] Sending connection context
0320 02:57:48.934572 (+ 208us) client_negotiation.cc:240] Negotiation successful
0320 02:57:48.934885 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
I20260320 02:57:48.938133 25136 heartbeater.cc:344] Connected to a master server at 127.23.132.126:43669
I20260320 02:57:48.938661 25136 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:48.939383 25136 heartbeater.cc:507] Master 127.23.132.126:43669 requested a full tablet report, sending...
I20260320 02:57:48.940969 24669 ts_manager.cc:194] Registered new tserver with Master: 4d89b049c787452b81021e53564af253 (127.23.132.67:36771)
I20260320 02:57:48.941849 24669 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:41373
I20260320 02:57:48.953572 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:57:48.965251 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.957352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.23.132.126:43669)
0320 02:57:48.957666 (+ 314us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.957671 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.957830 (+ 159us) server_negotiation.cc:408] Connection header received
0320 02:57:48.958011 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.958015 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.958066 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.958147 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:48.959122 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.959609 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.960307 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.960522 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.961837 (+ 1315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:48.961858 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:48.961861 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:48.961887 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:48.963490 (+ 1603us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.964183 (+ 693us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.964186 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.964188 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.964233 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:48.964448 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:48.964451 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:48.964453 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:48.964636 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:48.964767 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.964966 (+ 199us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.965089 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260320 02:57:48.973039 24669 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:57:48.976079 24669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:48.994238 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.989649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:39991 (local address 127.0.0.1:35176)
0320 02:57:48.990154 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:48.990167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:48.990248 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:48.991121 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:48.991129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:48.991228 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:48.991692 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.991703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.992869 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.992873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:48.993682 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.993690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.993822 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.993887 (+ 65us) client_negotiation.cc:769] Sending connection context
0320 02:57:48.993981 (+ 94us) client_negotiation.cc:240] Negotiation successful
0320 02:57:48.994077 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":260,"threads_started":1}
I20260320 02:57:48.994467 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.988173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:36771 (local address 127.0.0.1:56834)
0320 02:57:48.988619 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:48.988655 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:48.988778 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:48.990993 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:48.990999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:48.991021 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:48.991329 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.991338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.993205 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.993209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:48.994110 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.994121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.994226 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.994244 (+ 18us) client_negotiation.cc:769] Sending connection context
0320 02:57:48.994290 (+ 46us) client_negotiation.cc:240] Negotiation successful
0320 02:57:48.994346 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":95,"threads_started":1}
I20260320 02:57:48.994896 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.989869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.23.132.66:39991)
0320 02:57:48.990418 (+ 549us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.990425 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.990444 (+ 19us) server_negotiation.cc:408] Connection header received
0320 02:57:48.990525 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.990530 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.990788 (+ 258us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.990904 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:48.991932 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.992737 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.993903 (+ 1166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.994448 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.994575 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.994642 (+ 67us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.994739 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":294,"threads_started":1}
I20260320 02:57:48.995200 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.988397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:43723 (local address 127.0.0.1:58988)
0320 02:57:48.989239 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:48.989253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:48.989326 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:48.991295 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:48.991299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:48.991318 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:48.991692 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.991703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.993691 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.993694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:48.994552 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:48.994562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.994653 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.994668 (+ 15us) client_negotiation.cc:769] Sending connection context
0320 02:57:48.995004 (+ 336us) client_negotiation.cc:240] Negotiation successful
0320 02:57:48.995086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":650,"threads_started":1}
I20260320 02:57:48.995950 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.990037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.23.132.67:36771)
0320 02:57:48.990464 (+ 427us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.990470 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.990486 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:57:48.990547 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.990551 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.990724 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.990882 (+ 158us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:48.991451 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.993064 (+ 1613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.994876 (+ 1812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.995452 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.995551 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.995613 (+ 62us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.995801 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":161,"threads_started":1}
I20260320 02:57:48.996827 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:48.988990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.23.132.65:43723)
0320 02:57:48.990520 (+ 1530us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:48.990527 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:48.990548 (+ 21us) server_negotiation.cc:408] Connection header received
0320 02:57:48.990634 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:48.990639 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:48.991056 (+ 417us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:48.991205 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:48.992155 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.993570 (+ 1415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:48.995881 (+ 2311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:48.996411 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:48.996508 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:48.996589 (+ 81us) server_negotiation.cc:300] Negotiation successful
0320 02:57:48.996696 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":302,"threads_started":1}
I20260320 02:57:48.998857 24932 tablet_service.cc:1511] Processing CreateTablet for tablet fa3ef3aaec5b4f07be78d385d0b43052 (DEFAULT_TABLE table=test-table [id=1d41a8b8a9084e1c93e314cff9e6044d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:48.998982 25062 tablet_service.cc:1511] Processing CreateTablet for tablet fa3ef3aaec5b4f07be78d385d0b43052 (DEFAULT_TABLE table=test-table [id=1d41a8b8a9084e1c93e314cff9e6044d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:48.999208 24794 tablet_service.cc:1511] Processing CreateTablet for tablet fa3ef3aaec5b4f07be78d385d0b43052 (DEFAULT_TABLE table=test-table [id=1d41a8b8a9084e1c93e314cff9e6044d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:49.000200 24932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa3ef3aaec5b4f07be78d385d0b43052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:49.000274 25062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa3ef3aaec5b4f07be78d385d0b43052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:49.000274 24794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa3ef3aaec5b4f07be78d385d0b43052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:49.009020 25155 tablet_bootstrap.cc:492] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: Bootstrap starting.
I20260320 02:57:49.009485 25154 tablet_bootstrap.cc:492] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: Bootstrap starting.
I20260320 02:57:49.009485 25153 tablet_bootstrap.cc:492] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: Bootstrap starting.
I20260320 02:57:49.011924 25155 tablet_bootstrap.cc:654] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:49.012950 25155 log.cc:826] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:49.014218 25153 tablet_bootstrap.cc:654] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:49.014218 25154 tablet_bootstrap.cc:654] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:49.015292 25154 log.cc:826] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:49.015292 25153 log.cc:826] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:49.017148 25155 tablet_bootstrap.cc:492] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: No bootstrap required, opened a new log
I20260320 02:57:49.017282 25153 tablet_bootstrap.cc:492] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: No bootstrap required, opened a new log
I20260320 02:57:49.017390 25155 ts_tablet_manager.cc:1403] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260320 02:57:49.017418 25153 ts_tablet_manager.cc:1403] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260320 02:57:49.018219 25154 tablet_bootstrap.cc:492] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: No bootstrap required, opened a new log
I20260320 02:57:49.018414 25154 ts_tablet_manager.cc:1403] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260320 02:57:49.022423 25155 raft_consensus.cc:359] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.022831 25155 raft_consensus.cc:385] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:49.023000 25155 raft_consensus.cc:740] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db4d790d7ff4e0bb86032cf65c7c1fa, State: Initialized, Role: FOLLOWER
I20260320 02:57:49.023577 25155 consensus_queue.cc:260] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.023672 25154 raft_consensus.cc:359] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.023671 25153 raft_consensus.cc:359] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.024479 25153 raft_consensus.cc:385] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:49.024492 25154 raft_consensus.cc:385] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:49.024544 25153 raft_consensus.cc:740] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dad660241b94db09a81a025e653b877, State: Initialized, Role: FOLLOWER
I20260320 02:57:49.024559 25154 raft_consensus.cc:740] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89b049c787452b81021e53564af253, State: Initialized, Role: FOLLOWER
I20260320 02:57:49.025035 25153 consensus_queue.cc:260] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.025206 25155 ts_tablet_manager.cc:1434] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260320 02:57:49.025593 24863 heartbeater.cc:499] Master 127.23.132.126:43669 was elected leader, sending a full tablet report...
I20260320 02:57:49.025522 25154 consensus_queue.cc:260] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.025923 25153 ts_tablet_manager.cc:1434] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260320 02:57:49.028690 25000 heartbeater.cc:499] Master 127.23.132.126:43669 was elected leader, sending a full tablet report...
I20260320 02:57:49.029038 25154 ts_tablet_manager.cc:1434] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260320 02:57:49.029418 25136 heartbeater.cc:499] Master 127.23.132.126:43669 was elected leader, sending a full tablet report...
W20260320 02:57:49.071774 24864 tablet.cc:2379] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:57:49.135996 25001 tablet.cc:2379] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:57:49.168653 25137 tablet.cc:2379] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:57:49.279258 25161 raft_consensus.cc:493] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:49.279469 25161 raft_consensus.cc:515] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.280730 25161 leader_election.cc:290] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8db4d790d7ff4e0bb86032cf65c7c1fa (127.23.132.65:43723), 1dad660241b94db09a81a025e653b877 (127.23.132.66:39991)
I20260320 02:57:49.284262 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.280986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:43723 (local address 127.23.132.67:36335)
0320 02:57:49.281154 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:49.281308 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:49.281421 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:49.281764 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:49.281767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:49.281811 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:49.282145 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:49.282154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.283247 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.283251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:49.283941 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:49.283950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.284047 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.284071 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:57:49.284103 (+ 32us) client_negotiation.cc:240] Negotiation successful
0320 02:57:49.284156 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260320 02:57:49.284934 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.281125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:36335 (local address 127.23.132.65:43723)
0320 02:57:49.281418 (+ 293us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:49.281422 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:49.281438 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:57:49.281521 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:49.281525 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:49.281575 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:49.281656 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:49.282290 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.282971 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.284124 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.284605 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.284704 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:49.284758 (+ 54us) server_negotiation.cc:300] Negotiation successful
0320 02:57:49.284820 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260320 02:57:49.286005 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.281322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:39991 (local address 127.23.132.67:43513)
0320 02:57:49.281744 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:49.281759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:49.281875 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:49.282141 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:49.282144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:49.282161 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:49.282470 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:49.282478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.283994 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.283998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:49.284977 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:49.284987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.285637 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.285657 (+ 20us) client_negotiation.cc:769] Sending connection context
0320 02:57:49.285744 (+ 87us) client_negotiation.cc:240] Negotiation successful
0320 02:57:49.285841 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":135,"threads_started":1}
I20260320 02:57:49.286165 24817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa3ef3aaec5b4f07be78d385d0b43052" candidate_uuid: "4d89b049c787452b81021e53564af253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" is_pre_election: true
I20260320 02:57:49.286252 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.281722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:43513 (local address 127.23.132.66:39991)
0320 02:57:49.281896 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:49.281900 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:49.281913 (+ 13us) server_negotiation.cc:408] Connection header received
0320 02:57:49.281964 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:49.281968 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:49.282015 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:49.282094 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:49.282643 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.283412 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.285132 (+ 1720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.285529 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.285572 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:49.285867 (+ 295us) server_negotiation.cc:300] Negotiation successful
0320 02:57:49.285948 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260320 02:57:49.286474 24817 raft_consensus.cc:2468] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89b049c787452b81021e53564af253 in term 0.
I20260320 02:57:49.287114 24953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa3ef3aaec5b4f07be78d385d0b43052" candidate_uuid: "4d89b049c787452b81021e53564af253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dad660241b94db09a81a025e653b877" is_pre_election: true
I20260320 02:57:49.287374 24953 raft_consensus.cc:2468] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89b049c787452b81021e53564af253 in term 0.
I20260320 02:57:49.288034 25026 leader_election.cc:304] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89b049c787452b81021e53564af253, 8db4d790d7ff4e0bb86032cf65c7c1fa; no voters:
I20260320 02:57:49.288725 25161 raft_consensus.cc:2804] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:49.288811 25161 raft_consensus.cc:493] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:49.288841 25161 raft_consensus.cc:3060] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:49.290222 25161 raft_consensus.cc:515] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.291116 25161 leader_election.cc:290] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [CANDIDATE]: Term 1 election: Requested vote from peers 8db4d790d7ff4e0bb86032cf65c7c1fa (127.23.132.65:43723), 1dad660241b94db09a81a025e653b877 (127.23.132.66:39991)
I20260320 02:57:49.292683 24817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa3ef3aaec5b4f07be78d385d0b43052" candidate_uuid: "4d89b049c787452b81021e53564af253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa"
I20260320 02:57:49.292826 24817 raft_consensus.cc:3060] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:49.293145 24953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa3ef3aaec5b4f07be78d385d0b43052" candidate_uuid: "4d89b049c787452b81021e53564af253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dad660241b94db09a81a025e653b877"
I20260320 02:57:49.293294 24953 raft_consensus.cc:3060] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:49.294150 24817 raft_consensus.cc:2468] T fa3ef3aaec5b4f07be78d385d0b43052 P 8db4d790d7ff4e0bb86032cf65c7c1fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89b049c787452b81021e53564af253 in term 1.
I20260320 02:57:49.294503 25026 leader_election.cc:304] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89b049c787452b81021e53564af253, 8db4d790d7ff4e0bb86032cf65c7c1fa; no voters:
I20260320 02:57:49.294678 25161 raft_consensus.cc:2804] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:49.295074 25161 raft_consensus.cc:697] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [term 1 LEADER]: Becoming Leader. State: Replica: 4d89b049c787452b81021e53564af253, State: Running, Role: LEADER
I20260320 02:57:49.295203 24953 raft_consensus.cc:2468] T fa3ef3aaec5b4f07be78d385d0b43052 P 1dad660241b94db09a81a025e653b877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89b049c787452b81021e53564af253 in term 1.
I20260320 02:57:49.295424 25161 consensus_queue.cc:237] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } }
I20260320 02:57:49.299619 24669 catalog_manager.cc:5671] T fa3ef3aaec5b4f07be78d385d0b43052 P 4d89b049c787452b81021e53564af253 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d89b049c787452b81021e53564af253 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "4d89b049c787452b81021e53564af253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db4d790d7ff4e0bb86032cf65c7c1fa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 43723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d89b049c787452b81021e53564af253" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dad660241b94db09a81a025e653b877" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 39991 } health_report { overall_health: UNKNOWN } } }
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:57:49.380236 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.368169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.23.132.65:43723)
0320 02:57:49.368485 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:49.368488 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:49.368502 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:57:49.368542 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:49.368545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:49.368596 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:49.368777 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:49.370234 (+ 1457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.371389 (+ 1155us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.372706 (+ 1317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.372991 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.375483 (+ 2492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:49.375518 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:49.375531 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:49.375559 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:49.377729 (+ 2170us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:49.378489 (+ 760us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:49.378495 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:49.378496 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:49.378560 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:49.378933 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:49.378936 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:49.378938 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:49.379135 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:49.379300 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:49.379957 (+ 657us) server_negotiation.cc:300] Negotiation successful
0320 02:57:49.380089 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975469, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975469, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:57:49.422199 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.410192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.23.132.65:43723)
0320 02:57:49.410324 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:49.410328 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:49.411932 (+ 1604us) server_negotiation.cc:408] Connection header received
0320 02:57:49.411987 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:49.411990 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:49.412042 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:49.412122 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:49.413377 (+ 1255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.414135 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.415385 (+ 1250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.415775 (+ 390us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.418401 (+ 2626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:49.418425 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:49.418427 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:49.418460 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:49.420273 (+ 1813us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:49.420760 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:49.420763 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:49.420765 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:49.420815 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:49.421298 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:49.421301 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:49.421302 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:49.421456 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:49.421561 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:49.421964 (+ 403us) server_negotiation.cc:300] Negotiation successful
0320 02:57:49.422073 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:57:49.456259 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:49.444952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.23.132.65:43723)
0320 02:57:49.445138 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:49.445142 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:49.445154 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:57:49.445193 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:49.445196 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:49.445238 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:49.445310 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:49.446586 (+ 1276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.447135 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:49.448089 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:49.448278 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:49.452098 (+ 3820us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:49.452120 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:49.452123 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:49.452156 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:49.454333 (+ 2177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:49.454750 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:49.454754 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:49.454755 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:49.454800 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:49.455018 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:49.455021 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:49.455023 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:49.455357 (+ 334us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:49.455495 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:49.455969 (+ 474us) server_negotiation.cc:300] Negotiation successful
0320 02:57:49.456110 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260320 02:57:49.458141 24776 tablet_service.cc:3018] Rejecting scan request for tablet fa3ef3aaec5b4f07be78d385d0b43052: Uninitialized: clean time has not yet been initialized
I20260320 02:57:49.461464 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24733
I20260320 02:57:49.477007 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24870
I20260320 02:57:49.484628 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25007
I20260320 02:57:49.495136 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 24636
2026-03-20T02:57:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3954 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[25177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[25177](info): set up 2 sockets
Mar 20 02:57:49 dist-test-slave-v3hj krb5kdc[25177](info): commencing operation
krb5kdc: starting...
W20260320 02:57:51.644438 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.124s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:57:51 dist-test-slave-v3hj krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:57:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:51.851995 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39359
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:51.987373 25193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:51.987627 25193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:51.987716 25193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:51.991290 25193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:57:51.991377 25193 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260320 02:57:51.991402 25193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:51.991430 25193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:51.991456 25193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:57:51.991487 25193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:57:51.997771 25193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:39359
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:51.999190 25193 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:52.000336 25193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:52.008119 25198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:52.008126 25199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:52.009132 25201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:52.009498 25193 server_base.cc:1061] running on GCE node
I20260320 02:57:52.010764 25193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:52.012162 25193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:52.013523 25193 hybrid_clock.cc:648] HybridClock initialized: now 1773975472013455 us; error 105 us; skew 500 ppm
Mar 20 02:57:52 dist-test-slave-v3hj krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:52.018028 25193 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:57:52.021028 25193 webserver.cc:492] Webserver started at http://127.23.132.126:42681/ using document root <none> and password file <none>
I20260320 02:57:52.022228 25193 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:52.022454 25193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:52.022728 25193 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:52.024845 25193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "b5f00085e5de41679e668e3b42661f4b"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "7ba6ad614f1c6c9a82542d902fbc4f0e"
server_key_iv: "09ec958c5851aacae86cb2233e0a0f15"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.026091 25193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "b5f00085e5de41679e668e3b42661f4b"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "7ba6ad614f1c6c9a82542d902fbc4f0e"
server_key_iv: "09ec958c5851aacae86cb2233e0a0f15"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.031761 25193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260320 02:57:52.035578 25208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:52.038069 25193 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:57:52.038480 25193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "b5f00085e5de41679e668e3b42661f4b"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "7ba6ad614f1c6c9a82542d902fbc4f0e"
server_key_iv: "09ec958c5851aacae86cb2233e0a0f15"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.038861 25193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:52.054142 25193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:52.058357 25193 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:52.058576 25193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:52.070770 25193 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:39359
I20260320 02:57:52.070976 25270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:39359 every 8 connection(s)
I20260320 02:57:52.071986 25193 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:57:52.075315 25271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:52.080778 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25193
I20260320 02:57:52.080804 25271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: Bootstrap starting.
I20260320 02:57:52.081211 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:57:52.082136 24081 external_mini_cluster.cc:1468] Setting key 518c874b653646b0a87e07ba05966524
I20260320 02:57:52.083634 25271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:52.084606 25271 log.cc:826] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:52.088423 25271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: No bootstrap required, opened a new log
Mar 20 02:57:52 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:52.093606 25271 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f00085e5de41679e668e3b42661f4b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39359 } }
I20260320 02:57:52.093876 25271 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:52.093950 25271 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f00085e5de41679e668e3b42661f4b, State: Initialized, Role: FOLLOWER
I20260320 02:57:52.094987 25271 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f00085e5de41679e668e3b42661f4b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39359 } }
I20260320 02:57:52.095206 25271 raft_consensus.cc:399] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:57:52.095270 25271 raft_consensus.cc:493] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:57:52.095402 25271 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:52.097154 25271 raft_consensus.cc:515] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f00085e5de41679e668e3b42661f4b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39359 } }
I20260320 02:57:52.097741 25271 leader_election.cc:304] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f00085e5de41679e668e3b42661f4b; no voters:
I20260320 02:57:52.098276 25271 leader_election.cc:290] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:57:52.098580 25276 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:52.098968 25276 raft_consensus.cc:697] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [term 1 LEADER]: Becoming Leader. State: Replica: b5f00085e5de41679e668e3b42661f4b, State: Running, Role: LEADER
I20260320 02:57:52.099454 25276 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f00085e5de41679e668e3b42661f4b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39359 } }
I20260320 02:57:52.100409 25271 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:57:52.101157 25277 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f00085e5de41679e668e3b42661f4b. Latest consensus state: current_term: 1 leader_uuid: "b5f00085e5de41679e668e3b42661f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f00085e5de41679e668e3b42661f4b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39359 } } }
I20260320 02:57:52.101295 25277 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:52.101629 25278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f00085e5de41679e668e3b42661f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f00085e5de41679e668e3b42661f4b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39359 } } }
I20260320 02:57:52.101732 25278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:52.102556 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.083529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.23.132.126:39359)
0320 02:57:52.083940 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:52.083949 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:52.083983 (+ 34us) server_negotiation.cc:408] Connection header received
0320 02:57:52.084536 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:52.084551 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:52.084838 (+ 287us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:52.085135 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:52.086824 (+ 1689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.087584 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.088725 (+ 1141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.088936 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.093130 (+ 4194us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:52.093260 (+ 130us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:52.093281 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:52.093338 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:52.098040 (+ 4702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.099062 (+ 1022us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.099071 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.099078 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.099217 (+ 139us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.099511 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.099514 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.099516 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.100014 (+ 498us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:52.100211 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:52.100572 (+ 361us) server_negotiation.cc:300] Negotiation successful
0320 02:57:52.100864 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
W20260320 02:57:52.108748 25292 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:57:52.109019 25292 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:57:52.109320 25280 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:57:52.111546 25280 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:57:52.119601 25280 catalog_manager.cc:1357] Generated new cluster ID: 04c6676c7a90432e85f4f42d5914a02f
I20260320 02:57:52.119717 25280 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:57:52.158927 25280 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:57:52.160254 25280 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:57:52.181061 25280 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: Generated new TSK 0
I20260320 02:57:52.182449 25280 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:52.268246 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39359
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:52.414341 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:52.414592 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:52.414680 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:52.418546 25299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:52.418639 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:52.418723 25299 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:57:52.418790 25299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:57:52.424315 25299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:52.425556 25299 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:52.426481 25299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:52.437364 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:52.437366 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:52.437805 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:52.438616 25299 server_base.cc:1061] running on GCE node
I20260320 02:57:52.439133 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:52.439759 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:52.441004 25299 hybrid_clock.cc:648] HybridClock initialized: now 1773975472440950 us; error 70 us; skew 500 ppm
Mar 20 02:57:52 dist-test-slave-v3hj krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:52.444423 25299 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:57:52.446393 25299 webserver.cc:492] Webserver started at http://127.23.132.65:39773/ using document root <none> and password file <none>
I20260320 02:57:52.447394 25299 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:52.447458 25299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:52.447651 25299 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:52.449514 25299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "4b3f1889d6fa401d98a1f22043254afa"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "eb183c67bc897b13bd6332c810fb866d"
server_key_iv: "477f023a4f404e2c987f915f793db022"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.450150 25299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "4b3f1889d6fa401d98a1f22043254afa"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "eb183c67bc897b13bd6332c810fb866d"
server_key_iv: "477f023a4f404e2c987f915f793db022"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.454183 25299 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260320 02:57:52.457731 25314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:52.459422 25299 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:57:52.459570 25299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "4b3f1889d6fa401d98a1f22043254afa"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "eb183c67bc897b13bd6332c810fb866d"
server_key_iv: "477f023a4f404e2c987f915f793db022"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.459697 25299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:52.478247 25299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:52.482385 25299 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:52.482599 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:52.484359 25299 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:52.484428 25299 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:52.484485 25299 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:52.484520 25299 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:57:52 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:52.500778 25299 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:40767
I20260320 02:57:52.501015 25429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:40767 every 8 connection(s)
I20260320 02:57:52.503118 25299 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:57:52.505122 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.486453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:35005 (local address 127.23.132.126:39359)
0320 02:57:52.486688 (+ 235us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:52.486694 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:52.487452 (+ 758us) server_negotiation.cc:408] Connection header received
0320 02:57:52.488450 (+ 998us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:52.488455 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:52.488520 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:52.488575 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:52.491621 (+ 3046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.492429 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.493694 (+ 1265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.493916 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.498053 (+ 4137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:52.498084 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:52.498088 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:52.498130 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:52.500749 (+ 2619us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.501783 (+ 1034us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.501822 (+ 39us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.501868 (+ 46us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.501973 (+ 105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.502518 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.502522 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.502524 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.503072 (+ 548us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:52.503491 (+ 419us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:52.504646 (+ 1155us) server_negotiation.cc:300] Negotiation successful
0320 02:57:52.504925 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260320 02:57:52.506213 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.486482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.23.132.65:35005)
0320 02:57:52.487238 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:52.487282 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:52.488171 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:52.488997 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:52.489007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:52.490143 (+ 1136us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:52.491290 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.491305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.492700 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.492706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:52.493544 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.493560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.493880 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.494830 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:52.494865 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:52.497755 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:52.501011 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.501019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.501031 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.501478 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.502202 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.502205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.502207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.502353 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.503422 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:52.503430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:52.503736 (+ 306us) client_negotiation.cc:769] Sending connection context
0320 02:57:52.504556 (+ 820us) client_negotiation.cc:240] Negotiation successful
0320 02:57:52.505002 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":328,"threads_started":1}
I20260320 02:57:52.507632 25430 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39359
I20260320 02:57:52.507963 25430 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:52.509047 25430 heartbeater.cc:507] Master 127.23.132.126:39359 requested a full tablet report, sending...
I20260320 02:57:52.509168 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25299
I20260320 02:57:52.509440 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:57:52.510003 24081 external_mini_cluster.cc:1468] Setting key c132164d96a35139974918e23ad1ac47
I20260320 02:57:52.512616 25217 ts_manager.cc:194] Registered new tserver with Master: 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65:40767)
I20260320 02:57:52.514222 25217 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:35005
I20260320 02:57:52.521768 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.512377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:45861 (local address 127.23.132.126:39359)
0320 02:57:52.512505 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:52.512509 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:52.512815 (+ 306us) server_negotiation.cc:408] Connection header received
0320 02:57:52.512888 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:52.512891 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:52.512935 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:52.512999 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:52.513570 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.514791 (+ 1221us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.516218 (+ 1427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.516399 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.517455 (+ 1056us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:52.517495 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:52.517500 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:52.517549 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:52.519552 (+ 2003us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.520181 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.520186 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.520188 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.520267 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.520812 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.520815 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.520817 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.521004 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:52.521104 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:52.521482 (+ 378us) server_negotiation.cc:300] Negotiation successful
0320 02:57:52.521618 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260320 02:57:52.521804 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.512210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.23.132.65:45861)
0320 02:57:52.512676 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:52.512692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:52.512795 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:52.513035 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:52.513038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:52.513235 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:52.513453 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.513459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.514949 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.514953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:52.515833 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.515845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.515992 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.516593 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:52.516609 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:52.517177 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:52.519786 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.519790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.519791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.520008 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.520604 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.520607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.520609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.520688 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.521117 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:52.521121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:52.521284 (+ 163us) client_negotiation.cc:769] Sending connection context
0320 02:57:52.521392 (+ 108us) client_negotiation.cc:240] Negotiation successful
0320 02:57:52.521573 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:52.589159 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39359
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:52.724642 25441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:52.725203 25441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:52.725360 25441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:52.730352 25441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:52.730453 25441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:52.730510 25441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:57:52.730552 25441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:57:52.738235 25441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:52.739650 25441 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:52.740897 25441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:52.752390 25447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:52.752286 25446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:52.752394 25449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:52.753203 25441 server_base.cc:1061] running on GCE node
I20260320 02:57:52.753764 25441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:52.754509 25441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:52.755774 25441 hybrid_clock.cc:648] HybridClock initialized: now 1773975472755690 us; error 84 us; skew 500 ppm
Mar 20 02:57:52 dist-test-slave-v3hj krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:52.759279 25441 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:57:52.760512 25441 webserver.cc:492] Webserver started at http://127.23.132.66:38955/ using document root <none> and password file <none>
I20260320 02:57:52.761090 25441 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:52.761147 25441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:52.761322 25441 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:52.763671 25441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "ef6bc55601aa477fbaff95bb07eeb7c6"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "f94a2de983625a9c7cf64493bcc80a14"
server_key_iv: "3b5ff377be8593456f10000efd3661aa"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.764214 25441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "ef6bc55601aa477fbaff95bb07eeb7c6"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "f94a2de983625a9c7cf64493bcc80a14"
server_key_iv: "3b5ff377be8593456f10000efd3661aa"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.771827 25441 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260320 02:57:52.775550 25456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:52.777496 25441 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 02:57:52.777648 25441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "ef6bc55601aa477fbaff95bb07eeb7c6"
format_stamp: "Formatted at 2026-03-20 02:57:52 on dist-test-slave-v3hj"
server_key: "f94a2de983625a9c7cf64493bcc80a14"
server_key_iv: "3b5ff377be8593456f10000efd3661aa"
server_key_version: "encryptionkey@0"
I20260320 02:57:52.777860 25441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:52.791761 25441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:52.796101 25441 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:52.796295 25441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:52.797998 25441 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:52.798058 25441 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:52.798099 25441 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:52.798112 25441 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:57:52 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:52.813988 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.798972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.23.132.66:59851)
0320 02:57:52.799415 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:52.799455 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:52.800264 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:52.801095 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:52.801103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:52.801577 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:52.802354 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.802372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.803510 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.803514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:52.803903 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.803910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.804166 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.804739 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:52.804764 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:52.806536 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:52.809609 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.809618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.809632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.810001 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.811214 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.811218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.811220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.811361 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.812113 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:52.812119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:52.812466 (+ 347us) client_negotiation.cc:769] Sending connection context
0320 02:57:52.812795 (+ 329us) client_negotiation.cc:240] Negotiation successful
0320 02:57:52.813124 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260320 02:57:52.814245 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.798832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:59851 (local address 127.23.132.126:39359)
0320 02:57:52.798993 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:52.798998 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:52.799547 (+ 549us) server_negotiation.cc:408] Connection header received
0320 02:57:52.800562 (+ 1015us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:52.800567 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:52.800769 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:52.800899 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:52.802551 (+ 1652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.803389 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.804018 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.804183 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.806897 (+ 2714us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:52.806937 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:52.806942 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:52.806998 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:52.809445 (+ 2447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.810520 (+ 1075us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.810524 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.810526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.810606 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.811590 (+ 984us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.811596 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.811598 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.811932 (+ 334us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:52.812186 (+ 254us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:52.813954 (+ 1768us) server_negotiation.cc:300] Negotiation successful
0320 02:57:52.814105 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260320 02:57:52.818176 25441 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:40471
I20260320 02:57:52.818233 25576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:40471 every 8 connection(s)
I20260320 02:57:52.819450 25441 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:57:52.821235 25578 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39359
I20260320 02:57:52.821415 25578 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:52.822049 25578 heartbeater.cc:507] Master 127.23.132.126:39359 requested a full tablet report, sending...
I20260320 02:57:52.823326 25217 ts_manager.cc:194] Registered new tserver with Master: ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471)
I20260320 02:57:52.824154 25217 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:59851
I20260320 02:57:52.828922 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25441
I20260320 02:57:52.829053 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:57:52.829895 24081 external_mini_cluster.cc:1468] Setting key d36007c3a94870b656dc6eb996e2203e
I20260320 02:57:52.832598 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.819342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:58695 (local address 127.23.132.126:39359)
0320 02:57:52.819488 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:52.819493 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:52.819890 (+ 397us) server_negotiation.cc:408] Connection header received
0320 02:57:52.820041 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:52.820045 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:52.820195 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:52.820297 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:52.821013 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.821814 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.822879 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.823017 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.824117 (+ 1100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:52.824132 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:52.824134 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:52.824156 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:52.827077 (+ 2921us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.827520 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.827524 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.827526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.827570 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:52.828632 (+ 1062us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:52.828636 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:52.828805 (+ 169us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:52.829018 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:52.829567 (+ 549us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:52.831410 (+ 1843us) server_negotiation.cc:300] Negotiation successful
0320 02:57:52.831529 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260320 02:57:52.832635 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:52.819208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.23.132.66:58695)
0320 02:57:52.819772 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:52.819794 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:52.819924 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:52.820312 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:52.820315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:52.820502 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:52.820905 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.820916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.822046 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:52.822049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:52.822479 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:52.822490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:52.822591 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:52.823407 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:52.823422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:52.823824 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:52.827237 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.827241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.827242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.827418 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.828464 (+ 1046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:52.828468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:52.828471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:52.828528 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:52.830856 (+ 2328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:52.830861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:52.831026 (+ 165us) client_negotiation.cc:769] Sending connection context
0320 02:57:52.831335 (+ 309us) client_negotiation.cc:240] Negotiation successful
0320 02:57:52.831521 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:52.918584 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39359
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:53.072561 25584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:53.073061 25584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:53.073154 25584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:53.080124 25584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:53.080379 25584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:53.080603 25584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:57:53.080778 25584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:57:53.087224 25584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:53.088467 25584 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:53.089359 25584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:53.099879 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:53.099879 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:53.100448 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:53.100581 25584 server_base.cc:1061] running on GCE node
I20260320 02:57:53.101217 25584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:53.101882 25584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:53.103322 25584 hybrid_clock.cc:648] HybridClock initialized: now 1773975473103221 us; error 114 us; skew 500 ppm
Mar 20 02:57:53 dist-test-slave-v3hj krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:53.108223 25584 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:57:53.110751 25584 webserver.cc:492] Webserver started at http://127.23.132.67:40179/ using document root <none> and password file <none>
I20260320 02:57:53.111335 25584 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:53.111387 25584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:53.111554 25584 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:53.113343 25584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "b234da1c3bb54c15b9c723731161aebd"
format_stamp: "Formatted at 2026-03-20 02:57:53 on dist-test-slave-v3hj"
server_key: "39702e8971dcc1250c42cb0824f45d17"
server_key_iv: "5100db42d81bf9451e7a7340cc250161"
server_key_version: "encryptionkey@0"
I20260320 02:57:53.113848 25584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "b234da1c3bb54c15b9c723731161aebd"
format_stamp: "Formatted at 2026-03-20 02:57:53 on dist-test-slave-v3hj"
server_key: "39702e8971dcc1250c42cb0824f45d17"
server_key_iv: "5100db42d81bf9451e7a7340cc250161"
server_key_version: "encryptionkey@0"
I20260320 02:57:53.118014 25584 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260320 02:57:53.120338 25600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:53.122184 25584 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:53.122315 25584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "b234da1c3bb54c15b9c723731161aebd"
format_stamp: "Formatted at 2026-03-20 02:57:53 on dist-test-slave-v3hj"
server_key: "39702e8971dcc1250c42cb0824f45d17"
server_key_iv: "5100db42d81bf9451e7a7340cc250161"
server_key_version: "encryptionkey@0"
I20260320 02:57:53.122424 25584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:53.144285 25584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:53.154973 25584 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:53.155369 25584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:53.159154 25584 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:53.159237 25584 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:53.159297 25584 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:53.159349 25584 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:57:53 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975473, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:53.175896 25584 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:40431
I20260320 02:57:53.176206 25716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:40431 every 8 connection(s)
I20260320 02:57:53.177326 25584 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:57:53.179082 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25584
I20260320 02:57:53.179193 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:57:53.179634 24081 external_mini_cluster.cc:1468] Setting key 135a04a35bf6eb0f2668e1220ede773d
I20260320 02:57:53.180339 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.158940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:46285 (local address 127.23.132.126:39359)
0320 02:57:53.159107 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.159111 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.159858 (+ 747us) server_negotiation.cc:408] Connection header received
0320 02:57:53.161858 (+ 2000us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.161862 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.161919 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.162000 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:53.164220 (+ 2220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.165100 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.166079 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.166263 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.172399 (+ 6136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:53.172425 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:53.172428 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:53.172461 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:53.176530 (+ 4069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.177502 (+ 972us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.177515 (+ 13us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.177517 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.177643 (+ 126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.178455 (+ 812us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.178460 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.178462 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.178846 (+ 384us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:53.178971 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.179809 (+ 838us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.180169 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260320 02:57:53.181061 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.159099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.23.132.67:46285)
0320 02:57:53.159619 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.159703 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.161484 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.162223 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.162234 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.163061 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:53.163905 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.164010 (+ 105us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.165276 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.165280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.165924 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.165936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.166506 (+ 570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.167803 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:53.167861 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:53.172180 (+ 4319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:53.176753 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:53.176764 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:53.176775 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:53.177221 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:53.177972 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:53.177977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:53.177980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:53.178170 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:53.179030 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:53.179036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:53.179486 (+ 450us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.179755 (+ 269us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.180247 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":149,"threads_started":1}
I20260320 02:57:53.182964 25717 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39359
I20260320 02:57:53.183344 25717 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:53.184098 25717 heartbeater.cc:507] Master 127.23.132.126:39359 requested a full tablet report, sending...
I20260320 02:57:53.186523 25219 ts_manager.cc:194] Registered new tserver with Master: b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.187527 25219 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:46285
I20260320 02:57:53.200551 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:57:53.204816 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.188094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.23.132.67:37083)
0320 02:57:53.188498 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.188521 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.188758 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.191897 (+ 3139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.191902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.192389 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:53.192878 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.192893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.194621 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.194624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.195080 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.195089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.195562 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.196681 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:53.196700 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:53.197406 (+ 706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:53.200606 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:53.200611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:53.200614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:53.200909 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:53.201487 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:53.201492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:53.201494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:53.202782 (+ 1288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:53.204074 (+ 1292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:53.204078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:53.204305 (+ 227us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.204428 (+ 123us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.204614 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260320 02:57:53.204790 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.188128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:37083 (local address 127.23.132.126:39359)
0320 02:57:53.188245 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.188249 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.188768 (+ 519us) server_negotiation.cc:408] Connection header received
0320 02:57:53.190550 (+ 1782us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.190555 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.190616 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.191662 (+ 1046us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:53.193343 (+ 1681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.194367 (+ 1024us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.195574 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.195744 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.197552 (+ 1808us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:53.197572 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:53.197575 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:53.197604 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:53.200342 (+ 2738us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.201058 (+ 716us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.201062 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.201064 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.201121 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.203755 (+ 2634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.203759 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.203762 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.203953 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:53.204045 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.204508 (+ 463us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.204622 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Mar 20 02:57:53 dist-test-slave-v3hj krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975473, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 20 02:57:53 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975473, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:53.236658 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.222960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.23.132.126:39359)
0320 02:57:53.223518 (+ 558us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.223523 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.223549 (+ 26us) server_negotiation.cc:408] Connection header received
0320 02:57:53.224045 (+ 496us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.224053 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.224143 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.224450 (+ 307us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:53.225414 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.226502 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.228391 (+ 1889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.228652 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.232264 (+ 3612us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:53.232290 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:53.232293 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:53.232333 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:53.234251 (+ 1918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.235090 (+ 839us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.235093 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.235094 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.235148 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.235631 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.235634 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.235635 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.235829 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:53.235976 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.236357 (+ 381us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.236473 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260320 02:57:53.242084 25219 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:57:53.248894 25219 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:53.270349 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.263630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:40431 (local address 127.0.0.1:39632)
0320 02:57:53.265346 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.265358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.265968 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.267119 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.267123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.267147 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.267423 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.267430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.268871 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.268875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.269759 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.269768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.269922 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.269997 (+ 75us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.270092 (+ 95us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.270192 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":731,"threads_started":1}
I20260320 02:57:53.270991 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.264053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.23.132.67:40431)
0320 02:57:53.264445 (+ 392us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.264452 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.265429 (+ 977us) server_negotiation.cc:408] Connection header received
0320 02:57:53.266125 (+ 696us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.266131 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.266572 (+ 441us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.267104 (+ 532us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.267627 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.268754 (+ 1127us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.269940 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.270557 (+ 617us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.270695 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.270759 (+ 64us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.270848 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":107,"threads_started":1}
I20260320 02:57:53.274047 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.263317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40767 (local address 127.0.0.1:40056)
0320 02:57:53.264800 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.264842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.265001 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.266579 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.266585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.267162 (+ 577us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.267494 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.267502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.272500 (+ 4998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.272504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.273589 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.273600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.273734 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.273757 (+ 23us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.273840 (+ 83us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.273906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"spinlock_wait_cycles":1206400,"thread_start_us":1038,"threads_started":1}
I20260320 02:57:53.274885 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.265272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.23.132.65:40767)
0320 02:57:53.265545 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.265550 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.265571 (+ 21us) server_negotiation.cc:408] Connection header received
0320 02:57:53.265722 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.265730 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.266257 (+ 527us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.266413 (+ 156us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.267997 (+ 1584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.272157 (+ 4160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.273835 (+ 1678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.274495 (+ 660us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.274602 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.274687 (+ 85us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.274783 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260320 02:57:53.275432 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.263317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:40471 (local address 127.0.0.1:38618)
0320 02:57:53.265533 (+ 2216us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.265545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.265617 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.266590 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.266593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.267159 (+ 566us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.267494 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.267502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.274230 (+ 6728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.274234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.275075 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.275087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.275198 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.275215 (+ 17us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.275257 (+ 42us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.275311 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2160,"mutex_wait_us":1804,"spinlock_wait_cycles":3968,"thread_start_us":72,"threads_started":1}
I20260320 02:57:53.275079 25649 tablet_service.cc:1511] Processing CreateTablet for tablet 4cca71f1fe134efda128178e975abc12 (DEFAULT_TABLE table=test-table [id=1f8ebf92227849739cc1cc62d302136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:53.276366 25649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cca71f1fe134efda128178e975abc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.276801 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.265026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.23.132.66:40471)
0320 02:57:53.265818 (+ 792us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.265824 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.265846 (+ 22us) server_negotiation.cc:408] Connection header received
0320 02:57:53.266227 (+ 381us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.266233 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.266367 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.266481 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.267998 (+ 1517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.272157 (+ 4159us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.275640 (+ 3483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.276381 (+ 741us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.276496 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.276563 (+ 67us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.276658 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":106,"threads_started":1}
I20260320 02:57:53.277328 25363 tablet_service.cc:1511] Processing CreateTablet for tablet 4cca71f1fe134efda128178e975abc12 (DEFAULT_TABLE table=test-table [id=1f8ebf92227849739cc1cc62d302136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:53.278337 25363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cca71f1fe134efda128178e975abc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.279512 25506 tablet_service.cc:1511] Processing CreateTablet for tablet 4cca71f1fe134efda128178e975abc12 (DEFAULT_TABLE table=test-table [id=1f8ebf92227849739cc1cc62d302136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:53.281038 25506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cca71f1fe134efda128178e975abc12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.287654 25741 tablet_bootstrap.cc:492] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Bootstrap starting.
I20260320 02:57:53.291013 25741 tablet_bootstrap.cc:654] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.292773 25741 log.cc:826] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:53.292821 25742 tablet_bootstrap.cc:492] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Bootstrap starting.
I20260320 02:57:53.294783 25744 tablet_bootstrap.cc:492] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Bootstrap starting.
I20260320 02:57:53.298314 25742 tablet_bootstrap.cc:654] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.298413 25741 tablet_bootstrap.cc:492] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: No bootstrap required, opened a new log
I20260320 02:57:53.298661 25741 ts_tablet_manager.cc:1403] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260320 02:57:53.298857 25744 tablet_bootstrap.cc:654] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.300122 25744 log.cc:826] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:53.300217 25742 log.cc:826] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:53.304207 25744 tablet_bootstrap.cc:492] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: No bootstrap required, opened a new log
I20260320 02:57:53.304581 25741 raft_consensus.cc:359] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.304841 25742 tablet_bootstrap.cc:492] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: No bootstrap required, opened a new log
I20260320 02:57:53.304862 25741 raft_consensus.cc:385] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.304912 25741 raft_consensus.cc:740] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b234da1c3bb54c15b9c723731161aebd, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.305094 25742 ts_tablet_manager.cc:1403] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260320 02:57:53.305222 25744 ts_tablet_manager.cc:1403] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260320 02:57:53.305436 25741 consensus_queue.cc:260] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.306481 25741 ts_tablet_manager.cc:1434] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260320 02:57:53.307056 25717 heartbeater.cc:499] Master 127.23.132.126:39359 was elected leader, sending a full tablet report...
I20260320 02:57:53.309254 25742 raft_consensus.cc:359] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.309521 25742 raft_consensus.cc:385] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.309572 25742 raft_consensus.cc:740] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6bc55601aa477fbaff95bb07eeb7c6, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.310755 25742 consensus_queue.cc:260] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.312853 25578 heartbeater.cc:499] Master 127.23.132.126:39359 was elected leader, sending a full tablet report...
I20260320 02:57:53.312918 25742 ts_tablet_manager.cc:1434] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20260320 02:57:53.314680 25744 raft_consensus.cc:359] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.315398 25744 raft_consensus.cc:385] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.315469 25744 raft_consensus.cc:740] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3f1889d6fa401d98a1f22043254afa, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.316046 25744 consensus_queue.cc:260] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.317081 25744 ts_tablet_manager.cc:1434] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260320 02:57:53.318414 25430 heartbeater.cc:499] Master 127.23.132.126:39359 was elected leader, sending a full tablet report...
W20260320 02:57:53.321972 25579 tablet.cc:2379] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:57:53.366844 25749 raft_consensus.cc:493] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:53.367308 25749 raft_consensus.cc:515] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.368696 25749 leader_election.cc:290] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.371379 25748 raft_consensus.cc:493] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:53.371978 25748 raft_consensus.cc:515] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.373997 25748 leader_election.cc:290] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65:40767), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.375478 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.369316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:49901 (local address 127.23.132.67:40431)
0320 02:57:53.369562 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.369567 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.370033 (+ 466us) server_negotiation.cc:408] Connection header received
0320 02:57:53.370241 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.370244 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.370576 (+ 332us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.371203 (+ 627us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.372401 (+ 1198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.373381 (+ 980us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.374633 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.375112 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.375155 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.375319 (+ 164us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.375384 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260320 02:57:53.375486 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.369097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:40431 (local address 127.23.132.65:49901)
0320 02:57:53.369558 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.369574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.370122 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.371244 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.371248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.371273 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.371906 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.371918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.373551 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.373555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.374487 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.374498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.375182 (+ 684us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.375201 (+ 19us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.375273 (+ 72us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.375335 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":137,"threads_started":1}
I20260320 02:57:53.375680 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.369363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:40483 (local address 127.23.132.66:40471)
0320 02:57:53.369648 (+ 285us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.369654 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.370520 (+ 866us) server_negotiation.cc:408] Connection header received
0320 02:57:53.370633 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.370637 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.370698 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.371094 (+ 396us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.373063 (+ 1969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.373672 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.374800 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.375266 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.375303 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.375499 (+ 196us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.375567 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260320 02:57:53.376055 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.369097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:40471 (local address 127.23.132.65:40483)
0320 02:57:53.370389 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.370406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.370520 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.371102 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.371106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.371125 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.372835 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.372847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.373846 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.373849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.374654 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.374665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.375365 (+ 700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.375383 (+ 18us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.375849 (+ 466us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.375922 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":147,"thread_start_us":100,"threads_started":1}
I20260320 02:57:53.376525 25671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cca71f1fe134efda128178e975abc12" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd" is_pre_election: true
I20260320 02:57:53.376677 25527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cca71f1fe134efda128178e975abc12" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" is_pre_election: true
I20260320 02:57:53.376823 25527 raft_consensus.cc:2468] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 0.
I20260320 02:57:53.376824 25671 raft_consensus.cc:2468] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 0.
I20260320 02:57:53.377285 25317 leader_election.cc:304] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3f1889d6fa401d98a1f22043254afa, ef6bc55601aa477fbaff95bb07eeb7c6; no voters:
I20260320 02:57:53.377841 25749 raft_consensus.cc:2804] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:53.377928 25749 raft_consensus.cc:493] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:53.377972 25749 raft_consensus.cc:3060] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.378968 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.374253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:40431 (local address 127.23.132.66:51353)
0320 02:57:53.374924 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.374939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.375023 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.375750 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.375753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.375774 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.376022 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.376032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.377507 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.377510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.378549 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.378559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.378696 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.378720 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.378769 (+ 49us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.378834 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":72,"threads_started":1}
I20260320 02:57:53.379403 25749 raft_consensus.cc:515] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.379769 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.374737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:49907 (local address 127.23.132.65:40767)
0320 02:57:53.375101 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.375105 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.375558 (+ 453us) server_negotiation.cc:408] Connection header received
0320 02:57:53.375734 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.375738 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.375784 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.375831 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.376508 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.377010 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.378841 (+ 1831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.379324 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.379363 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.379602 (+ 239us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.379674 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260320 02:57:53.379841 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.374402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40767 (local address 127.23.132.66:49907)
0320 02:57:53.375492 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.375506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.375615 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.375968 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.375971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.375990 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:53.376364 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.376375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.377147 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.377150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.378499 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.378509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.379430 (+ 921us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.379447 (+ 17us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.379532 (+ 85us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.379605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":92,"threads_started":1}
I20260320 02:57:53.379906 25749 leader_election.cc:290] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 election: Requested vote from peers ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.380402 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.374596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:51353 (local address 127.23.132.67:40431)
0320 02:57:53.374966 (+ 370us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.374970 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.374984 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:57:53.375143 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.375146 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.375198 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.375280 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:53.376160 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.377386 (+ 1226us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.378735 (+ 1349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.379496 (+ 761us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.379621 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.379801 (+ 180us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.380274 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":103,"threads_started":1}
I20260320 02:57:53.380414 25384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cca71f1fe134efda128178e975abc12" candidate_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3f1889d6fa401d98a1f22043254afa" is_pre_election: true
I20260320 02:57:53.380640 25384 raft_consensus.cc:2393] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6bc55601aa477fbaff95bb07eeb7c6 in current term 1: Already voted for candidate 4b3f1889d6fa401d98a1f22043254afa in this term.
I20260320 02:57:53.380604 25527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cca71f1fe134efda128178e975abc12" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6"
I20260320 02:57:53.380620 25671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cca71f1fe134efda128178e975abc12" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd"
I20260320 02:57:53.380745 25527 raft_consensus.cc:3060] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.380796 25671 raft_consensus.cc:3060] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.381048 25670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cca71f1fe134efda128178e975abc12" candidate_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd" is_pre_election: true
I20260320 02:57:53.381652 25460 leader_election.cc:304] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6bc55601aa477fbaff95bb07eeb7c6; no voters: 4b3f1889d6fa401d98a1f22043254afa, b234da1c3bb54c15b9c723731161aebd
I20260320 02:57:53.382164 25671 raft_consensus.cc:2468] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 1.
I20260320 02:57:53.382164 25527 raft_consensus.cc:2468] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 1.
I20260320 02:57:53.382372 25748 raft_consensus.cc:2749] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260320 02:57:53.382509 25316 leader_election.cc:304] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3f1889d6fa401d98a1f22043254afa, b234da1c3bb54c15b9c723731161aebd; no voters:
I20260320 02:57:53.382848 25749 raft_consensus.cc:2804] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:53.383220 25749 raft_consensus.cc:697] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 1 LEADER]: Becoming Leader. State: Replica: 4b3f1889d6fa401d98a1f22043254afa, State: Running, Role: LEADER
I20260320 02:57:53.383543 25749 consensus_queue.cc:237] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.388317 25219 catalog_manager.cc:5671] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65). New cstate: current_term: 1 leader_uuid: "4b3f1889d6fa401d98a1f22043254afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260320 02:57:53.416267 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.411274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.23.132.65:40767)
0320 02:57:53.411534 (+ 260us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.411537 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.411549 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:57:53.411799 (+ 250us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.411805 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.412046 (+ 241us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.412192 (+ 146us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:53.413357 (+ 1165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.413893 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.414927 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.415082 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.415332 (+ 250us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.415820 (+ 488us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.415919 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.416088 (+ 169us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.416151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260320 02:57:53.423844 25527 raft_consensus.cc:1275] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Refusing update from remote peer 4b3f1889d6fa401d98a1f22043254afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:53.424346 25671 raft_consensus.cc:1275] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Refusing update from remote peer 4b3f1889d6fa401d98a1f22043254afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:53.424897 25749 consensus_queue.cc:1048] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:53.425532 25755 consensus_queue.cc:1048] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260320 02:57:53.428695 25718 tablet.cc:2379] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:57:53.438469 25758 mvcc.cc:204] Tried to move back new op lower bound from 7266203539134930944 to 7266203538981974016. Current Snapshot: MvccSnapshot[applied={T|T < 7266203539134930944}]
I20260320 02:57:53.440718 25760 mvcc.cc:204] Tried to move back new op lower bound from 7266203539134930944 to 7266203538981974016. Current Snapshot: MvccSnapshot[applied={T|T < 7266203539134930944}]
I20260320 02:57:53.454368 25217 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:57:53.454607 25217 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:57:53.460589 25217 catalog_manager.cc:5958] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: Sending DeleteTablet for 3 replicas of tablet 4cca71f1fe134efda128178e975abc12
I20260320 02:57:53.463049 25649 tablet_service.cc:1558] Processing DeleteTablet for tablet 4cca71f1fe134efda128178e975abc12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:53 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39632
I20260320 02:57:53.463234 25363 tablet_service.cc:1558] Processing DeleteTablet for tablet 4cca71f1fe134efda128178e975abc12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:53 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:40056
I20260320 02:57:53.464705 25768 tablet_replica.cc:333] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: stopping tablet replica
I20260320 02:57:53.465533 25217 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:57:53.466647 25217 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:53.467664 25768 raft_consensus.cc:2243] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:57:53.468398 25768 raft_consensus.cc:2272] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:53.464428 25506 tablet_service.cc:1558] Processing DeleteTablet for tablet 4cca71f1fe134efda128178e975abc12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:53 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38618
I20260320 02:57:53.469759 25769 tablet_replica.cc:333] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: stopping tablet replica
I20260320 02:57:53.470222 25769 raft_consensus.cc:2243] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:57:53.470870 25769 raft_consensus.cc:2272] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:53.473516 25768 ts_tablet_manager.cc:1916] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:53.474992 25770 tablet_replica.cc:333] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: stopping tablet replica
I20260320 02:57:53.475317 25770 raft_consensus.cc:2243] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:57:53.476768 25363 tablet_service.cc:1511] Processing CreateTablet for tablet 0a8b4231a8074a8babe46e89caab8cd5 (DEFAULT_TABLE table=test-table [id=0e079270eace49449b0e596529c9e88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:53.476828 25768 ts_tablet_manager.cc:1929] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:57:53.476927 25768 log.cc:1199] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/4cca71f1fe134efda128178e975abc12
I20260320 02:57:53.476950 25649 tablet_service.cc:1511] Processing CreateTablet for tablet 0a8b4231a8074a8babe46e89caab8cd5 (DEFAULT_TABLE table=test-table [id=0e079270eace49449b0e596529c9e88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:53.477073 25363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a8b4231a8074a8babe46e89caab8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.477174 25649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a8b4231a8074a8babe46e89caab8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.477552 25768 ts_tablet_manager.cc:1950] T 4cca71f1fe134efda128178e975abc12 P b234da1c3bb54c15b9c723731161aebd: Deleting consensus metadata
I20260320 02:57:53.478276 25506 tablet_service.cc:1511] Processing CreateTablet for tablet 0a8b4231a8074a8babe46e89caab8cd5 (DEFAULT_TABLE table=test-table [id=0e079270eace49449b0e596529c9e88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:53.478574 25506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a8b4231a8074a8babe46e89caab8cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.479565 25741 tablet_bootstrap.cc:492] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Bootstrap starting.
I20260320 02:57:53.480988 25741 tablet_bootstrap.cc:654] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.481333 25742 tablet_bootstrap.cc:492] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Bootstrap starting.
I20260320 02:57:53.481496 25744 tablet_bootstrap.cc:492] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Bootstrap starting.
I20260320 02:57:53.482411 25742 tablet_bootstrap.cc:654] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.482545 25744 tablet_bootstrap.cc:654] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.482551 25770 raft_consensus.cc:2272] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:53.484234 25741 tablet_bootstrap.cc:492] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: No bootstrap required, opened a new log
I20260320 02:57:53.484338 25741 ts_tablet_manager.cc:1403] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260320 02:57:53.484979 25741 raft_consensus.cc:359] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.485245 25769 ts_tablet_manager.cc:1916] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:53.485319 25741 raft_consensus.cc:385] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.485368 25741 raft_consensus.cc:740] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b234da1c3bb54c15b9c723731161aebd, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.485527 25770 ts_tablet_manager.cc:1916] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:53.485525 25741 consensus_queue.cc:260] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.488019 25741 ts_tablet_manager.cc:1434] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260320 02:57:53.488962 25744 tablet_bootstrap.cc:492] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: No bootstrap required, opened a new log
I20260320 02:57:53.489068 25744 ts_tablet_manager.cc:1403] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260320 02:57:53.489548 25744 raft_consensus.cc:359] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.489658 25744 raft_consensus.cc:385] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.489688 25744 raft_consensus.cc:740] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3f1889d6fa401d98a1f22043254afa, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.490845 25744 consensus_queue.cc:260] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.491130 25744 ts_tablet_manager.cc:1434] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260320 02:57:53.491165 25770 ts_tablet_manager.cc:1929] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:57:53.491508 25769 ts_tablet_manager.cc:1929] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:57:53.491595 25769 log.cc:1199] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/4cca71f1fe134efda128178e975abc12
I20260320 02:57:53.492476 25769 ts_tablet_manager.cc:1950] T 4cca71f1fe134efda128178e975abc12 P 4b3f1889d6fa401d98a1f22043254afa: Deleting consensus metadata
I20260320 02:57:53.493851 25210 catalog_manager.cc:5002] TS b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431): tablet 4cca71f1fe134efda128178e975abc12 (table test-table [id=1f8ebf92227849739cc1cc62d302136c]) successfully deleted
I20260320 02:57:53.494040 25212 catalog_manager.cc:5002] TS 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65:40767): tablet 4cca71f1fe134efda128178e975abc12 (table test-table [id=1f8ebf92227849739cc1cc62d302136c]) successfully deleted
I20260320 02:57:53.491272 25770 log.cc:1199] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/4cca71f1fe134efda128178e975abc12
I20260320 02:57:53.494488 25770 ts_tablet_manager.cc:1950] T 4cca71f1fe134efda128178e975abc12 P ef6bc55601aa477fbaff95bb07eeb7c6: Deleting consensus metadata
I20260320 02:57:53.497152 25742 tablet_bootstrap.cc:492] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: No bootstrap required, opened a new log
I20260320 02:57:53.497270 25742 ts_tablet_manager.cc:1403] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20260320 02:57:53.497470 25211 catalog_manager.cc:5002] TS ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471): tablet 4cca71f1fe134efda128178e975abc12 (table test-table [id=1f8ebf92227849739cc1cc62d302136c]) successfully deleted
I20260320 02:57:53.497771 25742 raft_consensus.cc:359] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.497946 25742 raft_consensus.cc:385] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.497987 25742 raft_consensus.cc:740] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6bc55601aa477fbaff95bb07eeb7c6, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.498116 25742 consensus_queue.cc:260] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.498634 25742 ts_tablet_manager.cc:1434] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260320 02:57:53.505378 25431 tablet.cc:2379] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:57:53.602283 25755 raft_consensus.cc:493] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:53.602458 25755 raft_consensus.cc:515] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.605180 25755 leader_election.cc:290] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.605722 25671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a8b4231a8074a8babe46e89caab8cd5" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd" is_pre_election: true
I20260320 02:57:53.606071 25671 raft_consensus.cc:2468] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 0.
I20260320 02:57:53.606284 25527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a8b4231a8074a8babe46e89caab8cd5" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" is_pre_election: true
I20260320 02:57:53.606393 25316 leader_election.cc:304] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3f1889d6fa401d98a1f22043254afa, b234da1c3bb54c15b9c723731161aebd; no voters:
I20260320 02:57:53.606451 25527 raft_consensus.cc:2468] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 0.
I20260320 02:57:53.606617 25755 raft_consensus.cc:2804] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:53.606973 25755 raft_consensus.cc:493] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:53.607046 25755 raft_consensus.cc:3060] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.608970 25755 raft_consensus.cc:515] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.610426 25755 leader_election.cc:290] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 election: Requested vote from peers ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.611078 25671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a8b4231a8074a8babe46e89caab8cd5" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd"
I20260320 02:57:53.611208 25671 raft_consensus.cc:3060] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.612498 25671 raft_consensus.cc:2468] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 1.
I20260320 02:57:53.612826 25527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a8b4231a8074a8babe46e89caab8cd5" candidate_uuid: "4b3f1889d6fa401d98a1f22043254afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6"
I20260320 02:57:53.612967 25527 raft_consensus.cc:3060] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.613391 25316 leader_election.cc:304] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3f1889d6fa401d98a1f22043254afa, b234da1c3bb54c15b9c723731161aebd; no voters:
I20260320 02:57:53.613642 25755 raft_consensus.cc:2804] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:53.613754 25755 raft_consensus.cc:697] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 1 LEADER]: Becoming Leader. State: Replica: 4b3f1889d6fa401d98a1f22043254afa, State: Running, Role: LEADER
I20260320 02:57:53.613936 25755 consensus_queue.cc:237] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.614444 25527 raft_consensus.cc:2468] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3f1889d6fa401d98a1f22043254afa in term 1.
I20260320 02:57:53.616283 25219 catalog_manager.cc:5671] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65). New cstate: current_term: 1 leader_uuid: "4b3f1889d6fa401d98a1f22043254afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } health_report { overall_health: UNKNOWN } } }
Mar 20 02:57:53 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975472, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:53.683992 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.672563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.23.132.126:39359)
0320 02:57:53.672815 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.672819 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.673132 (+ 313us) server_negotiation.cc:408] Connection header received
0320 02:57:53.673269 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.673273 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.673471 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.673571 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:53.674574 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.675078 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.675652 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.675810 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.679418 (+ 3608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:53.679440 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:53.679443 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:53.679468 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:53.681187 (+ 1719us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.681942 (+ 755us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.681946 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.681948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.682003 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:53.682816 (+ 813us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:53.682820 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:53.682821 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:53.683041 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:53.683157 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.683627 (+ 470us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.683762 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260320 02:57:53.684270 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.672718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39359 (local address 127.0.0.1:39622)
0320 02:57:53.673027 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.673043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.673175 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.673623 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.673627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.674229 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:53.674430 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.674437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.675188 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.675192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.675546 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.675552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.675659 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.676310 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:53.676323 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:53.679216 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:53.681346 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:53.681356 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:53.681360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:53.681675 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:53.682656 (+ 981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:53.682659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:53.682661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:53.682718 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:53.683262 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:53.683268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:53.683369 (+ 101us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.683466 (+ 97us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.683596 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
I20260320 02:57:53.691159 25219 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39622:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260320 02:57:53.692180 25219 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:53.698532 25363 tablet_service.cc:1511] Processing CreateTablet for tablet 760bbd6d1d3142d782059d35a2b9b1ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eea4fdca9b194ed9a3869ff704ee0fe7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:57:53.698670 25649 tablet_service.cc:1511] Processing CreateTablet for tablet 760bbd6d1d3142d782059d35a2b9b1ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eea4fdca9b194ed9a3869ff704ee0fe7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:57:53.698745 25506 tablet_service.cc:1511] Processing CreateTablet for tablet 760bbd6d1d3142d782059d35a2b9b1ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eea4fdca9b194ed9a3869ff704ee0fe7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:57:53.699080 25649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760bbd6d1d3142d782059d35a2b9b1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.699199 25363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760bbd6d1d3142d782059d35a2b9b1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.699219 25506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760bbd6d1d3142d782059d35a2b9b1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:53.702015 25744 tablet_bootstrap.cc:492] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: Bootstrap starting.
I20260320 02:57:53.703295 25741 tablet_bootstrap.cc:492] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: Bootstrap starting.
I20260320 02:57:53.704339 25744 tablet_bootstrap.cc:654] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.705152 25741 tablet_bootstrap.cc:654] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.706219 25742 tablet_bootstrap.cc:492] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: Bootstrap starting.
I20260320 02:57:53.707340 25742 tablet_bootstrap.cc:654] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:53.708506 25741 tablet_bootstrap.cc:492] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: No bootstrap required, opened a new log
I20260320 02:57:53.710036 25744 tablet_bootstrap.cc:492] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: No bootstrap required, opened a new log
I20260320 02:57:53.710160 25744 ts_tablet_manager.cc:1403] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260320 02:57:53.710649 25744 raft_consensus.cc:359] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.710815 25744 raft_consensus.cc:385] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.710842 25744 raft_consensus.cc:740] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3f1889d6fa401d98a1f22043254afa, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.710971 25744 consensus_queue.cc:260] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.711230 25744 ts_tablet_manager.cc:1434] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260320 02:57:53.708616 25741 ts_tablet_manager.cc:1403] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260320 02:57:53.713656 25742 tablet_bootstrap.cc:492] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: No bootstrap required, opened a new log
I20260320 02:57:53.713745 25742 ts_tablet_manager.cc:1403] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260320 02:57:53.714118 25741 raft_consensus.cc:359] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.714228 25741 raft_consensus.cc:385] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.714259 25741 raft_consensus.cc:740] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b234da1c3bb54c15b9c723731161aebd, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.714216 25742 raft_consensus.cc:359] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.714309 25742 raft_consensus.cc:385] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:53.714336 25742 raft_consensus.cc:740] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6bc55601aa477fbaff95bb07eeb7c6, State: Initialized, Role: FOLLOWER
I20260320 02:57:53.714389 25741 consensus_queue.cc:260] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.714437 25742 consensus_queue.cc:260] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.714660 25742 ts_tablet_manager.cc:1434] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:57:53.714674 25741 ts_tablet_manager.cc:1434] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260320 02:57:53.714876 25748 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.714973 25748 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.713677 25755 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.715068 25755 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.715163 25747 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.715269 25747 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.788448 25748 raft_consensus.cc:493] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:53.788609 25748 raft_consensus.cc:515] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.789348 25748 leader_election.cc:290] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65:40767), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.789851 25384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "760bbd6d1d3142d782059d35a2b9b1ca" candidate_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3f1889d6fa401d98a1f22043254afa" is_pre_election: true
I20260320 02:57:53.789850 25671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "760bbd6d1d3142d782059d35a2b9b1ca" candidate_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd" is_pre_election: true
I20260320 02:57:53.790035 25384 raft_consensus.cc:2468] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6bc55601aa477fbaff95bb07eeb7c6 in term 0.
I20260320 02:57:53.790629 25671 raft_consensus.cc:2468] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6bc55601aa477fbaff95bb07eeb7c6 in term 0.
I20260320 02:57:53.791270 25460 leader_election.cc:304] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3f1889d6fa401d98a1f22043254afa, ef6bc55601aa477fbaff95bb07eeb7c6; no voters:
I20260320 02:57:53.791724 25748 raft_consensus.cc:2804] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:53.791785 25748 raft_consensus.cc:493] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:53.791980 25748 raft_consensus.cc:3060] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.792975 25748 raft_consensus.cc:515] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.793323 25748 leader_election.cc:290] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [CANDIDATE]: Term 1 election: Requested vote from peers 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65:40767), b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431)
I20260320 02:57:53.793733 25384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "760bbd6d1d3142d782059d35a2b9b1ca" candidate_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3f1889d6fa401d98a1f22043254afa"
I20260320 02:57:53.793948 25671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "760bbd6d1d3142d782059d35a2b9b1ca" candidate_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b234da1c3bb54c15b9c723731161aebd"
I20260320 02:57:53.794088 25671 raft_consensus.cc:3060] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.794015 25384 raft_consensus.cc:3060] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:53.795535 25671 raft_consensus.cc:2468] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6bc55601aa477fbaff95bb07eeb7c6 in term 1.
I20260320 02:57:53.796162 25384 raft_consensus.cc:2468] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6bc55601aa477fbaff95bb07eeb7c6 in term 1.
I20260320 02:57:53.796162 25458 leader_election.cc:304] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b234da1c3bb54c15b9c723731161aebd, ef6bc55601aa477fbaff95bb07eeb7c6; no voters:
I20260320 02:57:53.796420 25748 raft_consensus.cc:2804] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:53.796662 25748 raft_consensus.cc:697] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 LEADER]: Becoming Leader. State: Replica: ef6bc55601aa477fbaff95bb07eeb7c6, State: Running, Role: LEADER
I20260320 02:57:53.797006 25748 consensus_queue.cc:237] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } }
I20260320 02:57:53.798519 25784 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: TxnStatusTablet state changed. Reason: New leader ef6bc55601aa477fbaff95bb07eeb7c6. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.798676 25784 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:57:53.798990 25786 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260320 02:57:53.799137 25786 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260320 02:57:53.800642 25217 catalog_manager.cc:5671] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66). New cstate: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260320 02:57:53.842532 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.829561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:40471 (local address 127.0.0.1:38622)
0320 02:57:53.829743 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.829762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.829973 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.830330 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.830334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.830345 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:53.830610 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.830618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.831747 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.831751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.832412 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.832423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.832541 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.832591 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.842153 (+ 9562us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.842189 (+ 36us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.842289 (+ 100us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.842397 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260320 02:57:53.842669 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.829682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.23.132.66:40471)
0320 02:57:53.829844 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.829848 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.829862 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:57:53.830144 (+ 282us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.830148 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.830210 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.830284 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:53.830768 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.831543 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.835246 (+ 3703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.835490 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.835568 (+ 78us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.836622 (+ 1054us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.836805 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.842357 (+ 5552us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.842518 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260320 02:57:53.847231 25671 raft_consensus.cc:1275] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Refusing update from remote peer ef6bc55601aa477fbaff95bb07eeb7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:53.847380 25384 raft_consensus.cc:1275] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Refusing update from remote peer ef6bc55601aa477fbaff95bb07eeb7c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:53.847836 25748 consensus_queue.cc:1048] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:53.849632 25784 consensus_queue.cc:1048] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:53.851373 25747 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: TxnStatusTablet state changed. Reason: New leader ef6bc55601aa477fbaff95bb07eeb7c6. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.851500 25747 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.854593 25755 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: TxnStatusTablet state changed. Reason: New leader ef6bc55601aa477fbaff95bb07eeb7c6. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.855154 25755 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.855927 25759 mvcc.cc:204] Tried to move back new op lower bound from 7266203540872024064 to 7266203540674490368. Current Snapshot: MvccSnapshot[applied={T|T < 7266203540872024064}]
I20260320 02:57:53.856115 25747 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.856202 25747 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P b234da1c3bb54c15b9c723731161aebd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.856822 25748 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.856930 25748 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:57:53.861670 25784 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.861850 25784 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P ef6bc55601aa477fbaff95bb07eeb7c6: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:57:53.863641 25755 tablet_replica.cc:442] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b3f1889d6fa401d98a1f22043254afa" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40767 } } peers { permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 } } peers { permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 } } }
I20260320 02:57:53.863853 25755 tablet_replica.cc:445] T 760bbd6d1d3142d782059d35a2b9b1ca P 4b3f1889d6fa401d98a1f22043254afa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:53.882848 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.879185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:51669 (local address 127.23.132.66:40471)
0320 02:57:53.879306 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.879311 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.879325 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:57:53.879382 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.879386 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.879442 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.879527 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:53.880003 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.880769 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.881779 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.882174 (+ 395us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.882234 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.882329 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.882506 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.882636 (+ 130us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.882699 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260320 02:57:53.883523 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.878801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:40471 (local address 127.23.132.65:51669)
0320 02:57:53.879109 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.879130 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.879250 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.879554 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.879557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.879567 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:53.879880 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.879888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.880981 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.880986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.881641 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.881662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.881783 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.881865 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.882529 (+ 664us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.882537 (+ 8us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.882617 (+ 80us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.883107 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260320 02:57:53.895680 25671 raft_consensus.cc:1275] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Refusing update from remote peer 4b3f1889d6fa401d98a1f22043254afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:53.895854 25527 raft_consensus.cc:1275] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Refusing update from remote peer 4b3f1889d6fa401d98a1f22043254afa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:53.896162 25755 consensus_queue.cc:1048] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b234da1c3bb54c15b9c723731161aebd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:53.897092 25749 consensus_queue.cc:1048] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6bc55601aa477fbaff95bb07eeb7c6" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 40471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:53.946791 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.934923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:43403 (local address 127.23.132.65:40767)
0320 02:57:53.938394 (+ 3471us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:53.938399 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:53.938416 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:57:53.938475 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:53.938479 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:53.938542 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:53.938646 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:53.941622 (+ 2976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.942477 (+ 855us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.945519 (+ 3042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.946061 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.946125 (+ 64us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.946230 (+ 105us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.946314 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:53.946487 (+ 173us) server_negotiation.cc:300] Negotiation successful
0320 02:57:53.946550 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3361,"thread_start_us":69,"threads_started":1}
I20260320 02:57:53.947379 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:53.934817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40767 (local address 127.23.132.66:43403)
0320 02:57:53.935201 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:53.937024 (+ 1823us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:53.937146 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:53.938791 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:53.938798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:53.938816 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:53.941410 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.941427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.942824 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:53.942831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:53.944550 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:53.944566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:53.945320 (+ 754us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:53.945390 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:53.946374 (+ 984us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:53.946386 (+ 12us) client_negotiation.cc:769] Sending connection context
0320 02:57:53.946468 (+ 82us) client_negotiation.cc:240] Negotiation successful
0320 02:57:53.946562 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":134,"threads_started":1}
I20260320 02:57:53.998142 25217 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:57:53.998366 25217 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:57:54.001418 25217 catalog_manager.cc:5958] T 00000000000000000000000000000000 P b5f00085e5de41679e668e3b42661f4b: Sending DeleteTablet for 3 replicas of tablet 0a8b4231a8074a8babe46e89caab8cd5
I20260320 02:57:54.002485 25363 tablet_service.cc:1558] Processing DeleteTablet for tablet 0a8b4231a8074a8babe46e89caab8cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:53 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:40056
I20260320 02:57:54.003150 25649 tablet_service.cc:1558] Processing DeleteTablet for tablet 0a8b4231a8074a8babe46e89caab8cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:53 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39632
I20260320 02:57:54.003372 25802 tablet_replica.cc:333] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: stopping tablet replica
I20260320 02:57:54.003526 25802 raft_consensus.cc:2243] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:57:54.004063 25802 raft_consensus.cc:2272] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:54.004877 25802 ts_tablet_manager.cc:1916] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:54.004695 25506 tablet_service.cc:1558] Processing DeleteTablet for tablet 0a8b4231a8074a8babe46e89caab8cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:53 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38618
I20260320 02:57:54.009198 25802 ts_tablet_manager.cc:1929] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:57:54.009315 25802 log.cc:1199] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/0a8b4231a8074a8babe46e89caab8cd5
I20260320 02:57:54.009619 25802 ts_tablet_manager.cc:1950] T 0a8b4231a8074a8babe46e89caab8cd5 P 4b3f1889d6fa401d98a1f22043254afa: Deleting consensus metadata
I20260320 02:57:54.010962 25212 catalog_manager.cc:5002] TS 4b3f1889d6fa401d98a1f22043254afa (127.23.132.65:40767): tablet 0a8b4231a8074a8babe46e89caab8cd5 (table test-table [id=0e079270eace49449b0e596529c9e88a]) successfully deleted
I20260320 02:57:54.012846 25803 tablet_replica.cc:333] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: stopping tablet replica
I20260320 02:57:54.013146 25803 raft_consensus.cc:2243] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:57:54.013453 25803 raft_consensus.cc:2272] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:54.013965 25804 tablet_replica.cc:333] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: stopping tablet replica
I20260320 02:57:54.014118 25804 raft_consensus.cc:2243] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:57:54.014251 25804 raft_consensus.cc:2272] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:54.015183 25803 ts_tablet_manager.cc:1916] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:54.015522 25804 ts_tablet_manager.cc:1916] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:54.017853 25803 ts_tablet_manager.cc:1929] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:57:54.017963 25803 log.cc:1199] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/0a8b4231a8074a8babe46e89caab8cd5
I20260320 02:57:54.018218 25803 ts_tablet_manager.cc:1950] T 0a8b4231a8074a8babe46e89caab8cd5 P b234da1c3bb54c15b9c723731161aebd: Deleting consensus metadata
I20260320 02:57:54.018978 25804 ts_tablet_manager.cc:1929] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:57:54.019073 25804 log.cc:1199] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/0a8b4231a8074a8babe46e89caab8cd5
I20260320 02:57:54.019363 25804 ts_tablet_manager.cc:1950] T 0a8b4231a8074a8babe46e89caab8cd5 P ef6bc55601aa477fbaff95bb07eeb7c6: Deleting consensus metadata
I20260320 02:57:54.019790 25210 catalog_manager.cc:5002] TS b234da1c3bb54c15b9c723731161aebd (127.23.132.67:40431): tablet 0a8b4231a8074a8babe46e89caab8cd5 (table test-table [id=0e079270eace49449b0e596529c9e88a]) successfully deleted
I20260320 02:57:54.020846 25211 catalog_manager.cc:5002] TS ef6bc55601aa477fbaff95bb07eeb7c6 (127.23.132.66:40471): tablet 0a8b4231a8074a8babe46e89caab8cd5 (table test-table [id=0e079270eace49449b0e596529c9e88a]) successfully deleted
Mar 20 02:57:54 dist-test-slave-v3hj krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975473, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:57:54.035532 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:54.012470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.23.132.65:40767)
0320 02:57:54.012599 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:54.012603 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:54.018886 (+ 6283us) server_negotiation.cc:408] Connection header received
0320 02:57:54.019208 (+ 322us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:54.019212 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:54.019276 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:54.019329 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:54.020519 (+ 1190us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:54.021425 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:54.022395 (+ 970us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:54.022618 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:54.028965 (+ 6347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:54.029009 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:54.029026 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:54.029070 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:54.032047 (+ 2977us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:54.033200 (+ 1153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:54.033208 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:54.033210 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:54.033290 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:54.033970 (+ 680us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:54.033973 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:54.033974 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:54.034178 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:54.034368 (+ 190us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:54.034630 (+ 262us) server_negotiation.cc:300] Negotiation successful
0320 02:57:54.034874 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260320 02:57:54.036165 25424 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076
I20260320 02:57:54.055668 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:54.039631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.23.132.126:39359)
0320 02:57:54.039910 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:54.039914 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:54.042231 (+ 2317us) server_negotiation.cc:408] Connection header received
0320 02:57:54.042638 (+ 407us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:54.042643 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:54.042789 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:54.042895 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:54.044139 (+ 1244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:54.044967 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:54.046265 (+ 1298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:54.046449 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:54.049756 (+ 3307us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:54.049810 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:54.049814 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:54.049853 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:54.052854 (+ 3001us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:54.054155 (+ 1301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:54.054160 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:54.054162 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:54.054223 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:54.054458 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:54.054461 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:54.054463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:54.054650 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:54.054777 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:54.055359 (+ 582us) server_negotiation.cc:300] Negotiation successful
0320 02:57:54.055484 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
W20260320 02:57:54.056638 25217 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39634
I20260320 02:57:54.058651 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25299
I20260320 02:57:54.071049 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25441
I20260320 02:57:54.083767 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25584
I20260320 02:57:54.092310 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25193
2026-03-20T02:57:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4596 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:57:54 dist-test-slave-v3hj krb5kdc[25811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:57:54 dist-test-slave-v3hj krb5kdc[25811](info): set up 2 sockets
Mar 20 02:57:54 dist-test-slave-v3hj krb5kdc[25811](info): commencing operation
krb5kdc: starting...
W20260320 02:57:56.248363 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.131s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:57:56 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:57:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:56.452028 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39733
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:56.580986 25828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:56.581254 25828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:56.581314 25828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:56.585151 25828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:57:56.585250 25828 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260320 02:57:56.585269 25828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:56.585289 25828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:56.585307 25828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:57:56.585323 25828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:57:56.591380 25828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:39733
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:56.592549 25828 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:56.593534 25828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260320 02:57:56.600548 25828 server_base.cc:1061] running on GCE node
W20260320 02:57:56.600688 25834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:56.601401 25833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:56.602059 25836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:56.603366 25828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:56.604610 25828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:56.608022 25828 hybrid_clock.cc:648] HybridClock initialized: now 1773975476607984 us; error 160 us; skew 500 ppm
Mar 20 02:57:56 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975476, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:56.612257 25828 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:57:56.614173 25828 webserver.cc:492] Webserver started at http://127.23.132.126:35603/ using document root <none> and password file <none>
I20260320 02:57:56.614845 25828 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:56.614914 25828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:56.615104 25828 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:56.616902 25828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "40d76999fd4b46b0893906374db6a6ef"
format_stamp: "Formatted at 2026-03-20 02:57:56 on dist-test-slave-v3hj"
server_key: "83064f5496669a320358f0e203d0664e"
server_key_iv: "917672113bd4ad2da807e3ff32182fe5"
server_key_version: "encryptionkey@0"
I20260320 02:57:56.617419 25828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "40d76999fd4b46b0893906374db6a6ef"
format_stamp: "Formatted at 2026-03-20 02:57:56 on dist-test-slave-v3hj"
server_key: "83064f5496669a320358f0e203d0664e"
server_key_iv: "917672113bd4ad2da807e3ff32182fe5"
server_key_version: "encryptionkey@0"
I20260320 02:57:56.621049 25828 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260320 02:57:56.624981 25843 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:56.626391 25828 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:56.626518 25828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "40d76999fd4b46b0893906374db6a6ef"
format_stamp: "Formatted at 2026-03-20 02:57:56 on dist-test-slave-v3hj"
server_key: "83064f5496669a320358f0e203d0664e"
server_key_iv: "917672113bd4ad2da807e3ff32182fe5"
server_key_version: "encryptionkey@0"
I20260320 02:57:56.626645 25828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:56.676980 25828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:56.683203 25828 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:56.683430 25828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:56.701169 25828 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:39733
I20260320 02:57:56.701615 25905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:39733 every 8 connection(s)
I20260320 02:57:56.703591 25828 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:57:56.710115 25906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:56.714028 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25828
I20260320 02:57:56.714175 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:57:56.714471 24081 external_mini_cluster.cc:1468] Setting key a92c657ebc4cb0182972dac829fa4c64
I20260320 02:57:56.718950 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Bootstrap starting.
I20260320 02:57:56.724356 25906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:56.725420 25906 log.cc:826] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Log is configured to *not* fsync() on all Append() calls
Mar 20 02:57:56 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:56.729738 25906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: No bootstrap required, opened a new log
I20260320 02:57:56.737350 25906 raft_consensus.cc:359] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76999fd4b46b0893906374db6a6ef" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39733 } }
I20260320 02:57:56.737658 25906 raft_consensus.cc:385] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:56.737727 25906 raft_consensus.cc:740] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d76999fd4b46b0893906374db6a6ef, State: Initialized, Role: FOLLOWER
I20260320 02:57:56.738341 25906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76999fd4b46b0893906374db6a6ef" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39733 } }
I20260320 02:57:56.738503 25906 raft_consensus.cc:399] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:57:56.738592 25906 raft_consensus.cc:493] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:57:56.738727 25906 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:56.740549 25906 raft_consensus.cc:515] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76999fd4b46b0893906374db6a6ef" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39733 } }
I20260320 02:57:56.741240 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:56.716125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.23.132.126:39733)
0320 02:57:56.716788 (+ 663us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:56.716806 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:56.717054 (+ 248us) server_negotiation.cc:408] Connection header received
0320 02:57:56.718357 (+ 1303us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:56.718387 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:56.718856 (+ 469us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:56.719256 (+ 400us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:56.720200 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:56.721441 (+ 1241us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:56.723636 (+ 2195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:56.723977 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:56.729931 (+ 5954us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:56.729965 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:56.729982 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:56.730035 (+ 53us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:56.736510 (+ 6475us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:56.737558 (+ 1048us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:56.737568 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:56.737575 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:56.737732 (+ 157us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:56.738118 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:56.738121 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:56.738123 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:56.738637 (+ 514us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:56.738881 (+ 244us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:56.739798 (+ 917us) server_negotiation.cc:300] Negotiation successful
0320 02:57:56.740172 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":146,"threads_started":1}
I20260320 02:57:56.741189 25906 leader_election.cc:304] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d76999fd4b46b0893906374db6a6ef; no voters:
I20260320 02:57:56.741873 25906 leader_election.cc:290] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:57:56.741971 25911 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:56.742127 25911 raft_consensus.cc:697] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [term 1 LEADER]: Becoming Leader. State: Replica: 40d76999fd4b46b0893906374db6a6ef, State: Running, Role: LEADER
I20260320 02:57:56.742604 25911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76999fd4b46b0893906374db6a6ef" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39733 } }
I20260320 02:57:56.743575 25906 sys_catalog.cc:565] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:57:56.744223 25911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d76999fd4b46b0893906374db6a6ef. Latest consensus state: current_term: 1 leader_uuid: "40d76999fd4b46b0893906374db6a6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76999fd4b46b0893906374db6a6ef" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39733 } } }
I20260320 02:57:56.744377 25911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [sys.catalog]: This master's current role is: LEADER
I20260320 02:57:56.745298 25912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40d76999fd4b46b0893906374db6a6ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d76999fd4b46b0893906374db6a6ef" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39733 } } }
I20260320 02:57:56.745404 25912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef [sys.catalog]: This master's current role is: LEADER
W20260320 02:57:56.751632 25926 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:57:56.752066 25926 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:57:56.752156 25916 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:57:56.753357 25916 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:57:56.761974 25916 catalog_manager.cc:1357] Generated new cluster ID: 471bce59649547649fccf9c2589fcae8
I20260320 02:57:56.762082 25916 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:57:56.774187 25916 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:57:56.776029 25916 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:57:56.794260 25916 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Generated new TSK 0
I20260320 02:57:56.795130 25916 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:56.888206 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39733
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:57.040625 25934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:57.040874 25934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:57.040940 25934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:57.045693 25934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:57.045816 25934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:57.045899 25934 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260320 02:57:57.045917 25934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:57:57.045964 25934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:57:57.051674 25934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:57.054374 25934 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:57.055573 25934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:57.063478 25940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:57.063701 25942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:57.063701 25939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:57.066464 25934 server_base.cc:1061] running on GCE node
I20260320 02:57:57.067487 25934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:57.068176 25934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:57.071774 25934 hybrid_clock.cc:648] HybridClock initialized: now 1773975477071652 us; error 145 us; skew 500 ppm
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:57.075577 25934 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:57:57.077128 25934 webserver.cc:492] Webserver started at http://127.23.132.65:35791/ using document root <none> and password file <none>
I20260320 02:57:57.078109 25934 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:57.078186 25934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:57.078413 25934 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:57.080687 25934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "1d8e3ba92c9543e0a0eba841e05ddf90"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "97a03c419e632d565d47672c959447ce"
server_key_iv: "3ab42f565fdc4abe3d744cd23c87b255"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.081215 25934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "1d8e3ba92c9543e0a0eba841e05ddf90"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "97a03c419e632d565d47672c959447ce"
server_key_iv: "3ab42f565fdc4abe3d744cd23c87b255"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.087245 25934 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260320 02:57:57.091754 25949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:57.093407 25934 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:57:57.093550 25934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "1d8e3ba92c9543e0a0eba841e05ddf90"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "97a03c419e632d565d47672c959447ce"
server_key_iv: "3ab42f565fdc4abe3d744cd23c87b255"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.093865 25934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:57.111756 25934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:57.117924 25934 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:57.118119 25934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:57.120096 25934 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:57.120162 25934 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:57.120215 25934 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:57.120236 25934 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:57.142848 25934 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:38953
I20260320 02:57:57.142930 26064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:38953 every 8 connection(s)
I20260320 02:57:57.143363 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.121505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:39335 (local address 127.23.132.126:39733)
0320 02:57:57.121705 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.121710 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.122562 (+ 852us) server_negotiation.cc:408] Connection header received
0320 02:57:57.124533 (+ 1971us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.124544 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.124655 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.127268 (+ 2613us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.130058 (+ 2790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.131302 (+ 1244us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.133361 (+ 2059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.133569 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.138427 (+ 4858us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.138454 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.138457 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.138490 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.140224 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.140956 (+ 732us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.140960 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.140962 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.141019 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.141601 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.141607 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.141609 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.142096 (+ 487us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.142202 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.142966 (+ 764us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.143224 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260320 02:57:57.148612 25934 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:57:57.148952 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.121947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.23.132.65:39335)
0320 02:57:57.122407 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.122447 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.123477 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.127426 (+ 3949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.127440 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.128325 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:57.129274 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.129298 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.131561 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.131566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.132918 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.132933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.133575 (+ 642us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.135236 (+ 1661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:57.135267 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:57.138201 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:57.140380 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.140388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.140406 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.140817 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.141116 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.141119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.141121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.141237 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.142280 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:57.142287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:57.142651 (+ 364us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.142910 (+ 259us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.146420 (+ 3510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260320 02:57:57.150112 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 25934
I20260320 02:57:57.150251 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:57:57.150512 24081 external_mini_cluster.cc:1468] Setting key bd8a166bb449077c776d4d06bfbe6de4
I20260320 02:57:57.153228 26065 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39733
I20260320 02:57:57.153450 26065 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:57.154219 26065 heartbeater.cc:507] Master 127.23.132.126:39733 requested a full tablet report, sending...
I20260320 02:57:57.156137 25857 ts_manager.cc:194] Registered new tserver with Master: 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953)
I20260320 02:57:57.158465 25857 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:39335
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:57:57.169505 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.155465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.23.132.65:39915)
0320 02:57:57.156526 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.156542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.156658 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.157150 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.157155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.157737 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:57.158182 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.158195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.159359 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.159365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.159989 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.160001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.160125 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.160913 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:57.160931 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:57.161328 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:57.166100 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.166106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.166108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.166571 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.167519 (+ 948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.167524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.167526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.167603 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.168550 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:57.168668 (+ 118us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:57.168880 (+ 212us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.169026 (+ 146us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.169248 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":89,"threads_started":1}
I20260320 02:57:57.169528 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.155545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:39915 (local address 127.23.132.126:39733)
0320 02:57:57.155691 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.155694 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.156687 (+ 993us) server_negotiation.cc:408] Connection header received
0320 02:57:57.156741 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.156744 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.156786 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.156855 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.158409 (+ 1554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.159175 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.160119 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.160308 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.161655 (+ 1347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.161678 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.161724 (+ 46us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.161750 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.165588 (+ 3838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.166749 (+ 1161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.166753 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.166755 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.167250 (+ 495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.167730 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.167734 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.167735 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.167982 (+ 247us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.168524 (+ 542us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.169185 (+ 661us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.169364 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:57.237483 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39733
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:57.378733 26076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:57.379102 26076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:57.379196 26076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:57.385311 26076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:57.385620 26076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:57.386026 26076 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260320 02:57:57.386081 26076 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:57:57.386161 26076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:57:57.393358 26076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:57.395140 26076 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:57.398146 26076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:57.408510 26082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:57.408511 26081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:57.408676 26084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:57.409147 26076 server_base.cc:1061] running on GCE node
I20260320 02:57:57.410269 26076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:57.410943 26076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:57.414599 26076 hybrid_clock.cc:648] HybridClock initialized: now 1773975477414544 us; error 66 us; skew 500 ppm
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:57.418334 26076 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:57:57.419564 26076 webserver.cc:492] Webserver started at http://127.23.132.66:35305/ using document root <none> and password file <none>
I20260320 02:57:57.420611 26076 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:57.421164 26076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:57.421752 26076 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:57.426087 26076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "5cca94d7d46d436fb34a2c52e08c6384"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "5480c0a3dc4cc86365cd8d67ab79de48"
server_key_iv: "48aaa4b0f4ade2b0a05d96519058a983"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.426729 26076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "5cca94d7d46d436fb34a2c52e08c6384"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "5480c0a3dc4cc86365cd8d67ab79de48"
server_key_iv: "48aaa4b0f4ade2b0a05d96519058a983"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.435737 26076 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260320 02:57:57.439760 26091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:57.442317 26076 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260320 02:57:57.442471 26076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "5cca94d7d46d436fb34a2c52e08c6384"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "5480c0a3dc4cc86365cd8d67ab79de48"
server_key_iv: "48aaa4b0f4ade2b0a05d96519058a983"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.442588 26076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:57.460301 26076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:57.467299 26076 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:57.468106 26076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:57.470165 26076 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:57.470238 26076 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:57.470293 26076 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:57.470314 26076 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:57.488415 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.471233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:58693 (local address 127.23.132.126:39733)
0320 02:57:57.471375 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.471379 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.472506 (+ 1127us) server_negotiation.cc:408] Connection header received
0320 02:57:57.474022 (+ 1516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.474026 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.474088 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.474342 (+ 254us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.476772 (+ 2430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.477321 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.478629 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.478825 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.482916 (+ 4091us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.482948 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.482952 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.482992 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.485503 (+ 2511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.486419 (+ 916us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.486423 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.486425 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.486484 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.487077 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.487081 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.487083 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.487305 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.487411 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.488095 (+ 684us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.488240 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260320 02:57:57.488442 26076 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:44221
I20260320 02:57:57.488488 26206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:44221 every 8 connection(s)
I20260320 02:57:57.489473 26076 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:57:57.489470 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.471458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.23.132.66:58693)
0320 02:57:57.472321 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.472369 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.473597 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.474499 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.474511 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.475889 (+ 1378us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:57.476583 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.476597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.477443 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.477447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.478093 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.478104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.478318 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.479026 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:57.479063 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:57.482440 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:57.485683 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.485691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.485706 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.486285 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.486635 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.486638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.486641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.486871 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.487455 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:57.487461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:57.487746 (+ 285us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.487992 (+ 246us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.488299 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":144,"threads_started":1}
I20260320 02:57:57.492311 26207 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39733
I20260320 02:57:57.492487 26207 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:57.493053 26207 heartbeater.cc:507] Master 127.23.132.126:39733 requested a full tablet report, sending...
I20260320 02:57:57.494486 25857 ts_manager.cc:194] Registered new tserver with Master: 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221)
I20260320 02:57:57.495137 25857 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:58693
I20260320 02:57:57.497383 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 26076
I20260320 02:57:57.497480 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:57:57.497663 24081 external_mini_cluster.cc:1468] Setting key 7eaaea89f666e2494fe7a74d8153f462
I20260320 02:57:57.504810 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.495206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.23.132.66:40013)
0320 02:57:57.495646 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.495660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.495857 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.496308 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.496312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.496520 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:57.496747 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.496755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.498480 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.498484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.498936 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.498946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.499048 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.499737 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:57.499754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:57.500268 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:57.502515 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.502519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.502520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.502693 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.503316 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.503319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.503320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.503376 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.504095 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:57.504099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:57.504208 (+ 109us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.504454 (+ 246us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.504629 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":109,"threads_started":1}
I20260320 02:57:57.504845 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.495282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:40013 (local address 127.23.132.126:39733)
0320 02:57:57.495407 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.495412 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.495826 (+ 414us) server_negotiation.cc:408] Connection header received
0320 02:57:57.495967 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.495971 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.496085 (+ 114us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.496277 (+ 192us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.497148 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.498364 (+ 1216us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.499040 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.499218 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.500404 (+ 1186us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.500421 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.500424 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.500539 (+ 115us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.502349 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.502894 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.502901 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.502903 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.503096 (+ 193us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.503487 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.503490 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.503493 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.503765 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.503876 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.504543 (+ 667us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.504670 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:57:57.571936 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39733
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:57:57.702276 26218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:57:57.702611 26218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:57:57.702719 26218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:57:57.707157 26218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:57:57.707274 26218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:57:57.707368 26218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260320 02:57:57.707396 26218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:57:57.707603 26218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:57:57.713765 26218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:57:57.715134 26218 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:57:57.716038 26218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:57:57.724915 26226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:57.724906 26224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:57:57.724920 26223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:57:57.725401 26218 server_base.cc:1061] running on GCE node
I20260320 02:57:57.725939 26218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:57:57.726550 26218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:57:57.727846 26218 hybrid_clock.cc:648] HybridClock initialized: now 1773975477727814 us; error 53 us; skew 500 ppm
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:57:57.733157 26218 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:57:57.735044 26218 webserver.cc:492] Webserver started at http://127.23.132.67:46557/ using document root <none> and password file <none>
I20260320 02:57:57.735638 26218 fs_manager.cc:362] Metadata directory not provided
I20260320 02:57:57.735700 26218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:57:57.735895 26218 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:57:57.738137 26218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "a68fefdb05574a3d9c2a94da84e571a0"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "902c533c1d8f88cc4556399e55faf9d8"
server_key_iv: "be8e2d414063be4e87369d987a71a4d3"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.738651 26218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "a68fefdb05574a3d9c2a94da84e571a0"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "902c533c1d8f88cc4556399e55faf9d8"
server_key_iv: "be8e2d414063be4e87369d987a71a4d3"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.743319 26218 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:57:57.746121 26233 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:57:57.747913 26218 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260320 02:57:57.748061 26218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "a68fefdb05574a3d9c2a94da84e571a0"
format_stamp: "Formatted at 2026-03-20 02:57:57 on dist-test-slave-v3hj"
server_key: "902c533c1d8f88cc4556399e55faf9d8"
server_key_iv: "be8e2d414063be4e87369d987a71a4d3"
server_key_version: "encryptionkey@0"
I20260320 02:57:57.748180 26218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:57:57.762638 26218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:57:57.767207 26218 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:57:57.767423 26218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:57:57.770351 26218 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:57:57.770543 26218 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260320 02:57:57.770653 26218 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:57:57.770720 26218 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:57.786916 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.770548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:59261 (local address 127.23.132.126:39733)
0320 02:57:57.770807 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.770811 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.772027 (+ 1216us) server_negotiation.cc:408] Connection header received
0320 02:57:57.773406 (+ 1379us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.773412 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.773499 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.773586 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.775374 (+ 1788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.775888 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.776599 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.776802 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.780216 (+ 3414us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.780242 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.780245 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.780275 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.782945 (+ 2670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.784013 (+ 1068us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.784018 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.784020 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.784082 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.784509 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.784514 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.784515 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.784835 (+ 320us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.784962 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.786201 (+ 1239us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.786401 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260320 02:57:57.788252 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.771125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.23.132.67:59261)
0320 02:57:57.771833 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.771900 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.773006 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.773918 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.773935 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.774515 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:57.775163 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.775178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.776043 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.776046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.776480 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.776489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.776717 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.777840 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:57.777910 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:57.780027 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:57.783153 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.783161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.783178 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.783580 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.784217 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.784221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.784223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.784351 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.784999 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:57.785005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:57.785667 (+ 662us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.786012 (+ 345us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.786399 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":168,"threads_started":1}
I20260320 02:57:57.788931 26218 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:46039
I20260320 02:57:57.789546 26348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:46039 every 8 connection(s)
I20260320 02:57:57.790395 26218 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:57:57.792816 26349 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39733
I20260320 02:57:57.793308 26349 heartbeater.cc:461] Registering TS with master...
I20260320 02:57:57.794409 26349 heartbeater.cc:507] Master 127.23.132.126:39733 requested a full tablet report, sending...
I20260320 02:57:57.796252 25857 ts_manager.cc:194] Registered new tserver with Master: a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:57:57.796861 25857 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:59261
I20260320 02:57:57.799851 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 26218
I20260320 02:57:57.800099 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:57:57.800592 24081 external_mini_cluster.cc:1468] Setting key ba06791637a5a2e66f7c13b47fd0d3f2
I20260320 02:57:57.805037 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:57:57.806097 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.793700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:52415 (local address 127.23.132.126:39733)
0320 02:57:57.793871 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.793876 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.794258 (+ 382us) server_negotiation.cc:408] Connection header received
0320 02:57:57.794365 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.794368 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.794416 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.794486 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.795494 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.796269 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.797047 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.797490 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.798246 (+ 756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.798264 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.798267 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.798291 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.801658 (+ 3367us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.802928 (+ 1270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.802932 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.802934 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.803051 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.803432 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.803435 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.803436 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.803602 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.803702 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.805757 (+ 2055us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.805929 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260320 02:57:57.808290 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.793560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.23.132.67:52415)
0320 02:57:57.794108 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.794123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.794257 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.794525 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.794528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.794907 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:57.795201 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.795209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.796420 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.796424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.796882 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.796891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.797002 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.797582 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:57.797597 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:57.798039 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:57.802291 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.802295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.802297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.802514 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.803241 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:57.803244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:57.803246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:57.803312 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:57.803706 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:57.803710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:57.805413 (+ 1703us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.805645 (+ 232us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.806005 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":328,"threads_started":1}
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 20 02:57:57 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:57.840018 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.826619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.23.132.126:39733)
0320 02:57:57.826830 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.826834 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.827122 (+ 288us) server_negotiation.cc:408] Connection header received
0320 02:57:57.827305 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.827309 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.827369 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.827466 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:57.828584 (+ 1118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.829222 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.830689 (+ 1467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.830957 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.834411 (+ 3454us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:57.834448 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:57.834464 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:57.834621 (+ 157us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:57.837342 (+ 2721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.838300 (+ 958us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.838303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.838305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.838361 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:57.838952 (+ 591us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:57.838955 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:57.838956 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:57.839139 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:57.839265 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.839738 (+ 473us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.839861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260320 02:57:57.844796 25857 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:57:57.850795 25857 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:57.877842 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.871652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:44221 (local address 127.0.0.1:38854)
0320 02:57:57.872401 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.872417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.872506 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.873536 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.873539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.873557 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:57.873857 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.873865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.876260 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.876264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.877274 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.877284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.877389 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.877462 (+ 73us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.877554 (+ 92us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.877661 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":450,"threads_started":1}
I20260320 02:57:57.878403 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.872161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.23.132.66:44221)
0320 02:57:57.873120 (+ 959us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.873127 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.873152 (+ 25us) server_negotiation.cc:408] Connection header received
0320 02:57:57.873235 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.873239 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.873389 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.873992 (+ 603us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:57.874160 (+ 168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.876100 (+ 1940us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.877429 (+ 1329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.877988 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.878120 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.878189 (+ 69us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.878282 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":117,"threads_started":1}
I20260320 02:57:57.878787 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.868872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:46039 (local address 127.0.0.1:39102)
0320 02:57:57.870809 (+ 1937us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.871223 (+ 414us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.871710 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.873230 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.873236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.873262 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:57.873574 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.873582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.876427 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.876431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.878404 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.878415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.878529 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.878546 (+ 17us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.878593 (+ 47us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.878653 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":140,"threads_started":1}
I20260320 02:57:57.879453 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.872836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38953 (local address 127.0.0.1:44954)
0320 02:57:57.873335 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:57.873350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:57.873436 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:57.873941 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:57.873944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:57.873973 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:57.874193 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.874202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.876698 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.876701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:57.879160 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:57.879170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.879265 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.879288 (+ 23us) client_negotiation.cc:769] Sending connection context
0320 02:57:57.879323 (+ 35us) client_negotiation.cc:240] Negotiation successful
0320 02:57:57.879370 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":98,"threads_started":1}
I20260320 02:57:57.879545 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.868993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.23.132.67:46039)
0320 02:57:57.869386 (+ 393us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.869392 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.872235 (+ 2843us) server_negotiation.cc:408] Connection header received
0320 02:57:57.872340 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.872346 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.872550 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.872689 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:57.873701 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.876287 (+ 2586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.878623 (+ 2336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.879128 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.879260 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.879325 (+ 65us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.879413 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":110,"threads_started":1}
I20260320 02:57:57.881412 26129 tablet_service.cc:1511] Processing CreateTablet for tablet 15935c84a0f94d5ab4fc0345de9200b6 (DEFAULT_TABLE table=test-table [id=a90b529d6fb749b4a7b959eb0f756adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:57.882408 26129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15935c84a0f94d5ab4fc0345de9200b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:57.883001 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:57.873239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.23.132.65:38953)
0320 02:57:57.873550 (+ 311us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:57.873555 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:57.873570 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:57:57.873644 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:57.873648 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:57.873814 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:57.873950 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:57.875542 (+ 1592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.876569 (+ 1027us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:57.879888 (+ 3319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:57.882532 (+ 2644us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:57.882645 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:57.882738 (+ 93us) server_negotiation.cc:300] Negotiation successful
0320 02:57:57.882835 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260320 02:57:57.883502 26283 tablet_service.cc:1511] Processing CreateTablet for tablet 15935c84a0f94d5ab4fc0345de9200b6 (DEFAULT_TABLE table=test-table [id=a90b529d6fb749b4a7b959eb0f756adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:57.884421 26283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15935c84a0f94d5ab4fc0345de9200b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:57.887858 25998 tablet_service.cc:1511] Processing CreateTablet for tablet 15935c84a0f94d5ab4fc0345de9200b6 (DEFAULT_TABLE table=test-table [id=a90b529d6fb749b4a7b959eb0f756adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:57.889737 26373 tablet_bootstrap.cc:492] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Bootstrap starting.
I20260320 02:57:57.889778 25998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15935c84a0f94d5ab4fc0345de9200b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:57.894500 26374 tablet_bootstrap.cc:492] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Bootstrap starting.
I20260320 02:57:57.894923 26373 tablet_bootstrap.cc:654] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:57.898186 26375 tablet_bootstrap.cc:492] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Bootstrap starting.
I20260320 02:57:57.898684 26373 log.cc:826] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:57.900940 26374 tablet_bootstrap.cc:654] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:57.901209 26375 tablet_bootstrap.cc:654] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:57.901990 26375 log.cc:826] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:57.902029 26374 log.cc:826] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Log is configured to *not* fsync() on all Append() calls
I20260320 02:57:57.903720 26373 tablet_bootstrap.cc:492] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: No bootstrap required, opened a new log
I20260320 02:57:57.903759 26375 tablet_bootstrap.cc:492] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: No bootstrap required, opened a new log
I20260320 02:57:57.903756 26374 tablet_bootstrap.cc:492] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: No bootstrap required, opened a new log
I20260320 02:57:57.903944 26375 ts_tablet_manager.cc:1403] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260320 02:57:57.903944 26374 ts_tablet_manager.cc:1403] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260320 02:57:57.903947 26373 ts_tablet_manager.cc:1403] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260320 02:57:57.906477 26373 raft_consensus.cc:359] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:57.906710 26373 raft_consensus.cc:385] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:57.906770 26373 raft_consensus.cc:740] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68fefdb05574a3d9c2a94da84e571a0, State: Initialized, Role: FOLLOWER
I20260320 02:57:57.907229 26373 consensus_queue.cc:260] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:57.907354 26374 raft_consensus.cc:359] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:57.907356 26375 raft_consensus.cc:359] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:57.907557 26375 raft_consensus.cc:385] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:57.907557 26374 raft_consensus.cc:385] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:57.907605 26375 raft_consensus.cc:740] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Initialized, Role: FOLLOWER
I20260320 02:57:57.907605 26374 raft_consensus.cc:740] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Initialized, Role: FOLLOWER
I20260320 02:57:57.908022 26375 consensus_queue.cc:260] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:57.907997 26374 consensus_queue.cc:260] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:57.908706 26065 heartbeater.cc:499] Master 127.23.132.126:39733 was elected leader, sending a full tablet report...
I20260320 02:57:57.909672 26375 ts_tablet_manager.cc:1434] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260320 02:57:57.909672 26374 ts_tablet_manager.cc:1434] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260320 02:57:57.910106 26207 heartbeater.cc:499] Master 127.23.132.126:39733 was elected leader, sending a full tablet report...
I20260320 02:57:57.908524 26373 ts_tablet_manager.cc:1434] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260320 02:57:57.911312 26349 heartbeater.cc:499] Master 127.23.132.126:39733 was elected leader, sending a full tablet report...
W20260320 02:57:57.991216 26208 tablet.cc:2379] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:57:58.041622 26350 tablet.cc:2379] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:57:58.152454 26066 tablet.cc:2379] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:57:58.156899 26381 raft_consensus.cc:493] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:58.157127 26381 raft_consensus.cc:515] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.158550 26381 leader_election.cc:290] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039), 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953)
I20260320 02:57:58.159658 26380 raft_consensus.cc:493] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:58.160137 26380 raft_consensus.cc:515] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.161876 26380 leader_election.cc:290] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039), 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221)
I20260320 02:57:58.164263 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.159032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:45703 (local address 127.23.132.67:46039)
0320 02:57:58.159252 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.159256 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.159353 (+ 97us) server_negotiation.cc:408] Connection header received
0320 02:57:58.159602 (+ 249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.159606 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.159752 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.159825 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:58.160591 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.161289 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.162940 (+ 1651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.163614 (+ 674us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.163657 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.163983 (+ 326us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.164085 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260320 02:57:58.164413 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.159450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:40399 (local address 127.23.132.65:38953)
0320 02:57:58.159573 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.159577 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.159590 (+ 13us) server_negotiation.cc:408] Connection header received
0320 02:57:58.159710 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.159714 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.159763 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.159974 (+ 211us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:58.160379 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.161169 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.162864 (+ 1695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.163552 (+ 688us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.163625 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.163983 (+ 358us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.164301 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260320 02:57:58.165024 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.159278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38953 (local address 127.23.132.66:40399)
0320 02:57:58.159494 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.159508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.159601 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.160010 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.160013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.160034 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:58.160241 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.160247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.161465 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.161471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.162707 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.162718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.163762 (+ 1044us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.163786 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.164632 (+ 846us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.164732 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260320 02:57:58.165232 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.158828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:46039 (local address 127.23.132.66:45703)
0320 02:57:58.159240 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.159269 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.159434 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.160092 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.160095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.160117 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:58.160408 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.160417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.161560 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.161563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.162797 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.162804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.163761 (+ 957us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.163785 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.165017 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.165122 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260320 02:57:58.166405 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0" is_pre_election: true
I20260320 02:57:58.166342 26019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" is_pre_election: true
I20260320 02:57:58.166692 26019 raft_consensus.cc:2468] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 0.
I20260320 02:57:58.166740 26303 raft_consensus.cc:2468] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 0.
I20260320 02:57:58.167990 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.161363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:46039 (local address 127.23.132.65:46881)
0320 02:57:58.162190 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.162204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.162320 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.162691 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.162694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.162710 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:58.162955 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.162962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.164596 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.164601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.166705 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.166715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.167480 (+ 765us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.167516 (+ 36us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.167576 (+ 60us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.167815 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":144,"threads_started":1}
I20260320 02:57:58.168076 26093 leader_election.cc:304] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cca94d7d46d436fb34a2c52e08c6384, a68fefdb05574a3d9c2a94da84e571a0; no voters:
I20260320 02:57:58.168591 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.161763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:53749 (local address 127.23.132.66:44221)
0320 02:57:58.161911 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.161915 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.162411 (+ 496us) server_negotiation.cc:408] Connection header received
0320 02:57:58.162568 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.162572 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.162616 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.162659 (+ 43us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:58.163400 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.164517 (+ 1117us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.166679 (+ 2162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.167938 (+ 1259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.168031 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.168405 (+ 374us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.168482 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260320 02:57:58.168895 26381 raft_consensus.cc:2804] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:58.168970 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.161362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:44221 (local address 127.23.132.65:53749)
0320 02:57:58.162342 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.162354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.162464 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.162777 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.162780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.162794 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:58.163023 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.163030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.164659 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.164662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.166457 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.166473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.168234 (+ 1761us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.168270 (+ 36us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.168760 (+ 490us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.168831 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":80,"thread_start_us":80,"threads_started":1}
I20260320 02:57:58.168979 26381 raft_consensus.cc:493] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:58.169030 26381 raft_consensus.cc:3060] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.169116 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.161941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:46881 (local address 127.23.132.67:46039)
0320 02:57:58.162214 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.162218 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.162416 (+ 198us) server_negotiation.cc:408] Connection header received
0320 02:57:58.162460 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.162463 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.162522 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.162592 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:58.163074 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.164385 (+ 1311us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.168208 (+ 3823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.168699 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.168757 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.168899 (+ 142us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.168965 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260320 02:57:58.169529 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384" is_pre_election: true
I20260320 02:57:58.169612 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0" is_pre_election: true
I20260320 02:57:58.169767 26303 raft_consensus.cc:2468] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 0.
I20260320 02:57:58.170212 26381 raft_consensus.cc:515] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.170487 26159 raft_consensus.cc:2393] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in current term 1: Already voted for candidate 5cca94d7d46d436fb34a2c52e08c6384 in this term.
I20260320 02:57:58.170584 26381 leader_election.cc:290] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 election: Requested vote from peers a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039), 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953)
I20260320 02:57:58.170872 25951 leader_election.cc:304] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, a68fefdb05574a3d9c2a94da84e571a0; no voters:
I20260320 02:57:58.171164 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0"
I20260320 02:57:58.171164 26019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90"
I20260320 02:57:58.171182 26380 raft_consensus.cc:2804] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:58.171298 26019 raft_consensus.cc:3060] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.171308 26303 raft_consensus.cc:3060] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.173271 26019 raft_consensus.cc:2468] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 1.
I20260320 02:57:58.173277 26303 raft_consensus.cc:2468] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 1.
I20260320 02:57:58.173491 26380 raft_consensus.cc:493] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:58.173586 26380 raft_consensus.cc:3060] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Advancing to term 2
I20260320 02:57:58.173609 26093 leader_election.cc:304] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, 5cca94d7d46d436fb34a2c52e08c6384; no voters:
I20260320 02:57:58.173977 26381 raft_consensus.cc:2804] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:58.174178 26381 raft_consensus.cc:697] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 LEADER]: Becoming Leader. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Running, Role: LEADER
I20260320 02:57:58.174863 26381 consensus_queue.cc:237] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.175323 26380 raft_consensus.cc:515] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.176460 26380 leader_election.cc:290] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 2 election: Requested vote from peers a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039), 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221)
I20260320 02:57:58.176883 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384"
I20260320 02:57:58.176848 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15935c84a0f94d5ab4fc0345de9200b6" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0"
I20260320 02:57:58.177014 26303 raft_consensus.cc:3060] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Advancing to term 2
I20260320 02:57:58.179700 26303 raft_consensus.cc:2468] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 2.
I20260320 02:57:58.179646 25857 catalog_manager.cc:5671] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66). New cstate: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } health_report { overall_health: UNKNOWN } } }
I20260320 02:57:58.180227 25951 leader_election.cc:304] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, a68fefdb05574a3d9c2a94da84e571a0; no voters: 5cca94d7d46d436fb34a2c52e08c6384
I20260320 02:57:58.180794 26380 raft_consensus.cc:2804] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 2 FOLLOWER]: Leader election won for term 2
I20260320 02:57:58.181059 26380 raft_consensus.cc:697] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 2 LEADER]: Becoming Leader. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Running, Role: LEADER
I20260320 02:57:58.181371 26380 consensus_queue.cc:237] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.188453 25857 catalog_manager.cc:5671] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 reported cstate change: term changed from 1 to 2, leader changed from 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66) to 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65). New cstate: current_term: 2 leader_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } health_report { overall_health: HEALTHY } } }
Mar 20 02:57:58 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975476, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:57:58.275151 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.259918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.23.132.126:39733)
0320 02:57:58.260076 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.260081 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.260216 (+ 135us) server_negotiation.cc:408] Connection header received
0320 02:57:58.260347 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.260350 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.260391 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.260459 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:57:58.261517 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.262123 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.263118 (+ 995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.263278 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.270260 (+ 6982us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:57:58.270286 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:57:58.270288 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:57:58.270414 (+ 126us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:57:58.272933 (+ 2519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:58.273860 (+ 927us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:58.273865 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:58.273866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:58.273927 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:57:58.274244 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:57:58.274247 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:57:58.274248 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:57:58.274422 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:57:58.274502 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.274904 (+ 402us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.275007 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260320 02:57:58.275256 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.259800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39733 (local address 127.0.0.1:56236)
0320 02:57:58.260094 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.260107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.260213 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.260584 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.260586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.261176 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:57:58.261370 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.261377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.262379 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.262385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.262921 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.262934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.263113 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.267394 (+ 4281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:57:58.267416 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:57:58.270001 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:57:58.273164 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:58.273184 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:58.273189 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:58.273558 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:58.274050 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:57:58.274053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:57:58.274055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:57:58.274113 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:57:58.274578 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:57:58.274585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:57:58.274694 (+ 109us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.274798 (+ 104us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.274940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
I20260320 02:57:58.280133 25857 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:56236:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260320 02:57:58.280880 25857 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:58.293867 25998 tablet_service.cc:1511] Processing CreateTablet for tablet c139137d44d840c9849a1dc8e6a166c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97ac3a9672d840aeaf8ba347eaf87a7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:57:58.293867 26129 tablet_service.cc:1511] Processing CreateTablet for tablet c139137d44d840c9849a1dc8e6a166c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97ac3a9672d840aeaf8ba347eaf87a7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:57:58.294204 26283 tablet_service.cc:1511] Processing CreateTablet for tablet c139137d44d840c9849a1dc8e6a166c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97ac3a9672d840aeaf8ba347eaf87a7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:57:58.294603 25998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c139137d44d840c9849a1dc8e6a166c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:58.294603 26129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c139137d44d840c9849a1dc8e6a166c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:58.294924 26283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c139137d44d840c9849a1dc8e6a166c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:58.297277 26373 tablet_bootstrap.cc:492] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: Bootstrap starting.
I20260320 02:57:58.297372 26374 tablet_bootstrap.cc:492] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: Bootstrap starting.
I20260320 02:57:58.298192 26375 tablet_bootstrap.cc:492] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: Bootstrap starting.
I20260320 02:57:58.298542 26373 tablet_bootstrap.cc:654] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:58.298643 26374 tablet_bootstrap.cc:654] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:58.299253 26375 tablet_bootstrap.cc:654] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:58.300139 26373 tablet_bootstrap.cc:492] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: No bootstrap required, opened a new log
I20260320 02:57:58.300207 26374 tablet_bootstrap.cc:492] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: No bootstrap required, opened a new log
I20260320 02:57:58.300239 26373 ts_tablet_manager.cc:1403] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:58.300293 26374 ts_tablet_manager.cc:1403] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260320 02:57:58.300611 26375 tablet_bootstrap.cc:492] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: No bootstrap required, opened a new log
I20260320 02:57:58.300690 26375 ts_tablet_manager.cc:1403] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260320 02:57:58.300767 26373 raft_consensus.cc:359] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.300798 26374 raft_consensus.cc:359] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.300886 26373 raft_consensus.cc:385] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:58.300902 26374 raft_consensus.cc:385] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:58.300920 26373 raft_consensus.cc:740] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68fefdb05574a3d9c2a94da84e571a0, State: Initialized, Role: FOLLOWER
I20260320 02:57:58.300931 26374 raft_consensus.cc:740] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Initialized, Role: FOLLOWER
I20260320 02:57:58.301138 26374 consensus_queue.cc:260] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.301065 26373 consensus_queue.cc:260] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.301411 26374 ts_tablet_manager.cc:1434] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:57:58.301390 26375 raft_consensus.cc:359] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.301447 26381 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.301512 26375 raft_consensus.cc:385] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:58.301527 26373 ts_tablet_manager.cc:1434] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:57:58.301546 26375 raft_consensus.cc:740] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Initialized, Role: FOLLOWER
I20260320 02:57:58.301573 26381 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.301661 26375 consensus_queue.cc:260] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.302065 26375 ts_tablet_manager.cc:1434] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260320 02:57:58.302078 26380 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.302232 26380 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.301771 26379 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.302909 26379 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.369674 26381 raft_consensus.cc:493] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:58.369927 26381 raft_consensus.cc:515] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.371460 26381 leader_election.cc:290] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953), a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:57:58.372208 26019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c139137d44d840c9849a1dc8e6a166c1" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" is_pre_election: true
I20260320 02:57:58.372560 26019 raft_consensus.cc:2468] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 0.
I20260320 02:57:58.372534 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c139137d44d840c9849a1dc8e6a166c1" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0" is_pre_election: true
I20260320 02:57:58.372741 26303 raft_consensus.cc:2468] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 0.
I20260320 02:57:58.373200 26093 leader_election.cc:304] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cca94d7d46d436fb34a2c52e08c6384, a68fefdb05574a3d9c2a94da84e571a0; no voters:
I20260320 02:57:58.373497 26381 raft_consensus.cc:2804] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:58.373564 26381 raft_consensus.cc:493] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:58.373611 26381 raft_consensus.cc:3060] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.375247 26381 raft_consensus.cc:515] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.376080 26381 leader_election.cc:290] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 election: Requested vote from peers 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953), a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:57:58.376839 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c139137d44d840c9849a1dc8e6a166c1" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0"
I20260320 02:57:58.376896 26019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c139137d44d840c9849a1dc8e6a166c1" candidate_uuid: "5cca94d7d46d436fb34a2c52e08c6384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90"
I20260320 02:57:58.377017 26303 raft_consensus.cc:3060] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.377027 26019 raft_consensus.cc:3060] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.377957 26019 raft_consensus.cc:2468] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 1.
I20260320 02:57:58.377957 26303 raft_consensus.cc:2468] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cca94d7d46d436fb34a2c52e08c6384 in term 1.
I20260320 02:57:58.378365 26093 leader_election.cc:304] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, 5cca94d7d46d436fb34a2c52e08c6384; no voters:
I20260320 02:57:58.378763 26381 raft_consensus.cc:2804] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:58.379047 26381 raft_consensus.cc:697] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 LEADER]: Becoming Leader. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Running, Role: LEADER
I20260320 02:57:58.379276 26381 consensus_queue.cc:237] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.380327 26387 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: TxnStatusTablet state changed. Reason: New leader 5cca94d7d46d436fb34a2c52e08c6384. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.380486 26387 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:57:58.380867 26401 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260320 02:57:58.381074 26401 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260320 02:57:58.381501 25858 catalog_manager.cc:5671] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66). New cstate: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } health_report { overall_health: UNKNOWN } } }
I20260320 02:57:58.398303 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.394829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.23.132.66:44221)
0320 02:57:58.394982 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.394987 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.395002 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:57:58.395376 (+ 374us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.395379 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.395443 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.395521 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.395868 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.396331 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.397126 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.397329 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.397382 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.397903 (+ 521us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.398009 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.398105 (+ 96us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.398157 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260320 02:57:58.398303 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.394692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:44221 (local address 127.0.0.1:38862)
0320 02:57:58.394942 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.394958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.395197 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.395550 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.395553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.395560 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:58.395761 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.395767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.396435 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.396438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.396941 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.396952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.397131 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.397184 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.398002 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.398018 (+ 16us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.398083 (+ 65us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.398169 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260320 02:57:58.405493 26019 raft_consensus.cc:1275] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Refusing update from remote peer 5cca94d7d46d436fb34a2c52e08c6384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:58.405565 26303 raft_consensus.cc:1275] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Refusing update from remote peer 5cca94d7d46d436fb34a2c52e08c6384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:58.407037 26381 consensus_queue.cc:1048] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:58.407467 26387 consensus_queue.cc:1048] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:58.415186 26380 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: TxnStatusTablet state changed. Reason: New leader 5cca94d7d46d436fb34a2c52e08c6384. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.415342 26380 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.415707 26379 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: TxnStatusTablet state changed. Reason: New leader 5cca94d7d46d436fb34a2c52e08c6384. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.415833 26379 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.420392 26379 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.420529 26379 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P a68fefdb05574a3d9c2a94da84e571a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.421106 26405 mvcc.cc:204] Tried to move back new op lower bound from 7266203559535632384 to 7266203559443644416. Current Snapshot: MvccSnapshot[applied={T|T < 7266203559535632384}]
I20260320 02:57:58.421581 26381 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.421891 26381 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:57:58.422062 26404 mvcc.cc:204] Tried to move back new op lower bound from 7266203559535632384 to 7266203559443644416. Current Snapshot: MvccSnapshot[applied={T|T < 7266203559535632384}]
I20260320 02:57:58.422569 26381 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.425235 26381 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 5cca94d7d46d436fb34a2c52e08c6384: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:57:58.433619 26406 mvcc.cc:204] Tried to move back new op lower bound from 7266203559535632384 to 7266203559443644416. Current Snapshot: MvccSnapshot[applied={T|T < 7266203559535632384}]
I20260320 02:57:58.434101 26380 tablet_replica.cc:442] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cca94d7d46d436fb34a2c52e08c6384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } }
I20260320 02:57:58.434224 26380 tablet_replica.cc:445] T c139137d44d840c9849a1dc8e6a166c1 P 1d8e3ba92c9543e0a0eba841e05ddf90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:57:58.458854 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.454305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.23.132.65:38953)
0320 02:57:58.454460 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.454465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.454611 (+ 146us) server_negotiation.cc:408] Connection header received
0320 02:57:58.454788 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.454792 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.454865 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.454985 (+ 120us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.455432 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.456223 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.457402 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.457630 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.457748 (+ 118us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.458388 (+ 640us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.458511 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.458637 (+ 126us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.458703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260320 02:57:58.476934 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.473593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:44315 (local address 127.23.132.66:44221)
0320 02:57:58.473871 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.473876 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.473970 (+ 94us) server_negotiation.cc:408] Connection header received
0320 02:57:58.474092 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.474096 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.474156 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.474250 (+ 94us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.474686 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.475420 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.476208 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.476399 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.476445 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.476532 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.476610 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.476748 (+ 138us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.476797 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260320 02:57:58.476966 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.473448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:44221 (local address 127.23.132.65:44315)
0320 02:57:58.473882 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.473897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.474000 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.474315 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.474318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.474341 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:58.474577 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.474583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.475568 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.475571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.476091 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.476100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.476220 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.476264 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.476646 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.476653 (+ 7us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.476721 (+ 68us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.476781 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260320 02:57:58.491600 26159 raft_consensus.cc:3055] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 LEADER]: Stepping down as leader of term 1
I20260320 02:57:58.491825 26159 raft_consensus.cc:740] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Running, Role: LEADER
I20260320 02:57:58.492010 26159 consensus_queue.cc:260] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } }
I20260320 02:57:58.492365 26159 raft_consensus.cc:3060] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Advancing to term 2
I20260320 02:57:58.492520 26303 raft_consensus.cc:1275] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 2 FOLLOWER]: Refusing update from remote peer 1d8e3ba92c9543e0a0eba841e05ddf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260320 02:57:58.494048 26389 consensus_queue.cc:1048] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:58.494402 26159 raft_consensus.cc:1275] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 2 FOLLOWER]: Refusing update from remote peer 1d8e3ba92c9543e0a0eba841e05ddf90: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260320 02:57:58.494920 26389 consensus_queue.cc:1048] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:58.495046 26389 consensus_queue.cc:1243] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Peer 5cca94d7d46d436fb34a2c52e08c6384 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260320 02:57:58.496698 26159 pending_rounds.cc:85] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Aborting all ops after (but not including) 0
I20260320 02:57:58.497066 26159 pending_rounds.cc:107] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260320 02:57:58.497188 26159 raft_consensus.cc:2889] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260320 02:57:58.550938 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.535978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38953 (local address 127.23.132.66:50555)
0320 02:57:58.544301 (+ 8323us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.544321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.544460 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.545128 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.545131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.545140 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:58.545588 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.545598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.546633 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.546637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.547807 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.547818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.548527 (+ 709us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.548574 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.548871 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.548879 (+ 8us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.548940 (+ 61us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.550782 (+ 1842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8150,"thread_start_us":110,"threads_started":1}
I20260320 02:57:58.550923 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.536083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:50555 (local address 127.23.132.65:38953)
0320 02:57:58.536218 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.536222 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.544497 (+ 8275us) server_negotiation.cc:408] Connection header received
0320 02:57:58.544618 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.544623 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.544719 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.545283 (+ 564us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.545833 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.546504 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.547958 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.548430 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.548697 (+ 267us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.548784 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.548848 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.549982 (+ 1134us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.550740 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260320 02:57:58.603032 25857 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:57:58.603194 25857 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:57:58.608987 25857 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Sending DeleteTablet for 3 replicas of tablet 15935c84a0f94d5ab4fc0345de9200b6
I20260320 02:57:58.609949 25998 tablet_service.cc:1558] Processing DeleteTablet for tablet 15935c84a0f94d5ab4fc0345de9200b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:58 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:44954
I20260320 02:57:58.609926 26129 tablet_service.cc:1558] Processing DeleteTablet for tablet 15935c84a0f94d5ab4fc0345de9200b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:58 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38854
I20260320 02:57:58.610536 25857 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260320 02:57:58.611652 26283 tablet_service.cc:1558] Processing DeleteTablet for tablet 15935c84a0f94d5ab4fc0345de9200b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:57:58 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39102
W20260320 02:57:58.611910 25857 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:57:58.613425 26426 tablet_replica.cc:333] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: stopping tablet replica
I20260320 02:57:58.613826 26426 raft_consensus.cc:2243] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 2 LEADER]: Raft consensus shutting down.
I20260320 02:57:58.614322 26426 raft_consensus.cc:2272] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:58.615787 26426 ts_tablet_manager.cc:1916] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:58.618211 26425 tablet_replica.cc:333] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: stopping tablet replica
I20260320 02:57:58.618582 26425 raft_consensus.cc:2243] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260320 02:57:58.618911 26425 raft_consensus.cc:2272] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:58.619861 25998 tablet_service.cc:1511] Processing CreateTablet for tablet e5d4bf7c97b4464b93ca2210cd718a95 (DEFAULT_TABLE table=test-table [id=597afe4f5fbc44f1aa3e9b7ff5556c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:58.619623 26129 tablet_service.cc:1511] Processing CreateTablet for tablet e5d4bf7c97b4464b93ca2210cd718a95 (DEFAULT_TABLE table=test-table [id=597afe4f5fbc44f1aa3e9b7ff5556c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:58.620162 25998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d4bf7c97b4464b93ca2210cd718a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:58.620236 26129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d4bf7c97b4464b93ca2210cd718a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:58.620831 26427 tablet_replica.cc:333] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: stopping tablet replica
I20260320 02:57:58.621097 26427 raft_consensus.cc:2243] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260320 02:57:58.621301 26427 raft_consensus.cc:2272] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260320 02:57:58.622655 26283 tablet_service.cc:1511] Processing CreateTablet for tablet e5d4bf7c97b4464b93ca2210cd718a95 (DEFAULT_TABLE table=test-table [id=597afe4f5fbc44f1aa3e9b7ff5556c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:57:58.622750 26375 tablet_bootstrap.cc:492] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Bootstrap starting.
I20260320 02:57:58.622948 26283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d4bf7c97b4464b93ca2210cd718a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:57:58.624684 26374 tablet_bootstrap.cc:492] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Bootstrap starting.
I20260320 02:57:58.625509 26375 tablet_bootstrap.cc:654] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:58.627082 26374 tablet_bootstrap.cc:654] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:58.628451 26373 tablet_bootstrap.cc:492] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Bootstrap starting.
I20260320 02:57:58.629710 26373 tablet_bootstrap.cc:654] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Neither blocks nor log segments found. Creating new log.
I20260320 02:57:58.633826 26374 tablet_bootstrap.cc:492] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: No bootstrap required, opened a new log
I20260320 02:57:58.634032 26374 ts_tablet_manager.cc:1403] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260320 02:57:58.634177 26425 ts_tablet_manager.cc:1916] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:58.634742 26374 raft_consensus.cc:359] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.635001 26374 raft_consensus.cc:385] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:58.635084 26374 raft_consensus.cc:740] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Initialized, Role: FOLLOWER
I20260320 02:57:58.635313 26374 consensus_queue.cc:260] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.638479 26427 ts_tablet_manager.cc:1916] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:57:58.638666 26374 ts_tablet_manager.cc:1434] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260320 02:57:58.641577 26427 ts_tablet_manager.cc:1929] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20260320 02:57:58.641672 26427 log.cc:1199] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/15935c84a0f94d5ab4fc0345de9200b6
I20260320 02:57:58.641958 26427 ts_tablet_manager.cc:1950] T 15935c84a0f94d5ab4fc0345de9200b6 P a68fefdb05574a3d9c2a94da84e571a0: Deleting consensus metadata
I20260320 02:57:58.642670 26373 tablet_bootstrap.cc:492] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: No bootstrap required, opened a new log
I20260320 02:57:58.642987 26373 ts_tablet_manager.cc:1403] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260320 02:57:58.643124 26375 tablet_bootstrap.cc:492] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: No bootstrap required, opened a new log
I20260320 02:57:58.643245 26375 ts_tablet_manager.cc:1403] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20260320 02:57:58.643965 26373 raft_consensus.cc:359] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.644152 26373 raft_consensus.cc:385] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:58.644186 26375 raft_consensus.cc:359] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.644315 26375 raft_consensus.cc:385] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:57:58.644371 26375 raft_consensus.cc:740] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Initialized, Role: FOLLOWER
I20260320 02:57:58.644510 26375 consensus_queue.cc:260] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.644539 26426 ts_tablet_manager.cc:1929] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20260320 02:57:58.644850 26375 ts_tablet_manager.cc:1434] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260320 02:57:58.646905 26373 raft_consensus.cc:740] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68fefdb05574a3d9c2a94da84e571a0, State: Initialized, Role: FOLLOWER
I20260320 02:57:58.647203 26373 consensus_queue.cc:260] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.644713 26426 log.cc:1199] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/15935c84a0f94d5ab4fc0345de9200b6
I20260320 02:57:58.648648 26426 ts_tablet_manager.cc:1950] T 15935c84a0f94d5ab4fc0345de9200b6 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting consensus metadata
I20260320 02:57:58.649027 26373 ts_tablet_manager.cc:1434] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260320 02:57:58.650599 25845 catalog_manager.cc:5002] TS 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953): tablet 15935c84a0f94d5ab4fc0345de9200b6 (table test-table [id=a90b529d6fb749b4a7b959eb0f756adc]) successfully deleted
I20260320 02:57:58.650902 26425 ts_tablet_manager.cc:1929] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.5
I20260320 02:57:58.651448 26425 log.cc:1199] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/15935c84a0f94d5ab4fc0345de9200b6
I20260320 02:57:58.652140 26425 ts_tablet_manager.cc:1950] T 15935c84a0f94d5ab4fc0345de9200b6 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting consensus metadata
I20260320 02:57:58.654403 25845 catalog_manager.cc:5002] TS a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039): tablet 15935c84a0f94d5ab4fc0345de9200b6 (table test-table [id=a90b529d6fb749b4a7b959eb0f756adc]) successfully deleted
I20260320 02:57:58.655911 25846 catalog_manager.cc:5002] TS 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221): tablet 15935c84a0f94d5ab4fc0345de9200b6 (table test-table [id=a90b529d6fb749b4a7b959eb0f756adc]) successfully deleted
I20260320 02:57:58.727958 26379 raft_consensus.cc:493] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:57:58.728128 26379 raft_consensus.cc:515] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.730048 26379 leader_election.cc:290] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953), 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221)
I20260320 02:57:58.735508 26433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.730537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:44221 (local address 127.23.132.67:35085)
0320 02:57:58.731688 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.731713 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.732118 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.733049 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.733053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.733074 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:58.733440 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.733452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.734500 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.734504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.735075 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.735081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.735196 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.735222 (+ 26us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.735272 (+ 50us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.735336 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":91,"thread_start_us":134,"threads_started":1}
I20260320 02:57:58.735898 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.730858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:35085 (local address 127.23.132.66:44221)
0320 02:57:58.731283 (+ 425us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.731288 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.731812 (+ 524us) server_negotiation.cc:408] Connection header received
0320 02:57:58.732270 (+ 458us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.732274 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.732529 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.732595 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:58.733831 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.734387 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.735193 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.735626 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.735671 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.735725 (+ 54us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.735788 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260320 02:57:58.736346 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d4bf7c97b4464b93ca2210cd718a95" candidate_uuid: "a68fefdb05574a3d9c2a94da84e571a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384" is_pre_election: true
I20260320 02:57:58.736507 26159 raft_consensus.cc:2468] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a68fefdb05574a3d9c2a94da84e571a0 in term 0.
I20260320 02:57:58.736866 26236 leader_election.cc:304] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cca94d7d46d436fb34a2c52e08c6384, a68fefdb05574a3d9c2a94da84e571a0; no voters:
I20260320 02:57:58.737118 26379 raft_consensus.cc:2804] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:57:58.737169 26379 raft_consensus.cc:493] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:57:58.737210 26379 raft_consensus.cc:3060] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.737838 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.730382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:55831 (local address 127.23.132.65:38953)
0320 02:57:58.730518 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.730522 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.730989 (+ 467us) server_negotiation.cc:408] Connection header received
0320 02:57:58.731632 (+ 643us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.731637 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.731721 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.731777 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:57:58.732894 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.733960 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.736196 (+ 2236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.737216 (+ 1020us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.737283 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.737601 (+ 318us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.737706 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260320 02:57:58.738032 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.730252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38953 (local address 127.23.132.67:55831)
0320 02:57:58.730762 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.730780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.731279 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.732145 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.732148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.732164 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:57:58.732765 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.732779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.734089 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.734093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.735589 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.735613 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.737392 (+ 1779us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.737418 (+ 26us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.737536 (+ 118us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.737665 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":78,"threads_started":1}
I20260320 02:57:58.738494 26019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d4bf7c97b4464b93ca2210cd718a95" candidate_uuid: "a68fefdb05574a3d9c2a94da84e571a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" is_pre_election: true
I20260320 02:57:58.738485 26379 raft_consensus.cc:515] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.738618 26019 raft_consensus.cc:2468] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a68fefdb05574a3d9c2a94da84e571a0 in term 0.
I20260320 02:57:58.738883 26379 leader_election.cc:290] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [CANDIDATE]: Term 1 election: Requested vote from peers 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953), 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221)
I20260320 02:57:58.739317 26019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d4bf7c97b4464b93ca2210cd718a95" candidate_uuid: "a68fefdb05574a3d9c2a94da84e571a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90"
I20260320 02:57:58.739317 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d4bf7c97b4464b93ca2210cd718a95" candidate_uuid: "a68fefdb05574a3d9c2a94da84e571a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384"
I20260320 02:57:58.739416 26019 raft_consensus.cc:3060] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.739418 26159 raft_consensus.cc:3060] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:57:58.740443 26019 raft_consensus.cc:2468] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a68fefdb05574a3d9c2a94da84e571a0 in term 1.
I20260320 02:57:58.740450 26159 raft_consensus.cc:2468] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a68fefdb05574a3d9c2a94da84e571a0 in term 1.
I20260320 02:57:58.740798 26236 leader_election.cc:304] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cca94d7d46d436fb34a2c52e08c6384, a68fefdb05574a3d9c2a94da84e571a0; no voters:
I20260320 02:57:58.740996 26379 raft_consensus.cc:2804] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:57:58.741633 26379 raft_consensus.cc:697] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 LEADER]: Becoming Leader. State: Replica: a68fefdb05574a3d9c2a94da84e571a0, State: Running, Role: LEADER
I20260320 02:57:58.741931 26379 consensus_queue.cc:237] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:57:58.745980 25857 catalog_manager.cc:5671] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "a68fefdb05574a3d9c2a94da84e571a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } health_report { overall_health: HEALTHY } } }
I20260320 02:57:58.795455 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.786692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.23.132.67:46039)
0320 02:57:58.787321 (+ 629us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.787324 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.787337 (+ 13us) server_negotiation.cc:408] Connection header received
0320 02:57:58.787417 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.787419 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.787712 (+ 293us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.788150 (+ 438us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.788941 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.790387 (+ 1446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.793560 (+ 3173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.793896 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.794085 (+ 189us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.794691 (+ 606us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.794844 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.795201 (+ 357us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.795272 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":147,"threads_started":1}
I20260320 02:57:58.814417 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.808465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:52667 (local address 127.23.132.66:44221)
0320 02:57:58.808673 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.808678 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.809071 (+ 393us) server_negotiation.cc:408] Connection header received
0320 02:57:58.809575 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.809580 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.810345 (+ 765us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.810447 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.811156 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.811895 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.812832 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.813526 (+ 694us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.813909 (+ 383us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.814011 (+ 102us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.814057 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.814209 (+ 152us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.814267 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260320 02:57:58.814850 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.808136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:44221 (local address 127.23.132.67:52667)
0320 02:57:58.808695 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.808716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.809396 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.810591 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.810594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.810607 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:58.811008 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.811019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.812033 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.812036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.812709 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.812717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.813629 (+ 912us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.813683 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.814119 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.814124 (+ 5us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.814623 (+ 499us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.814710 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":146,"threads_started":1}
I20260320 02:57:58.824517 26159 raft_consensus.cc:1275] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Refusing update from remote peer a68fefdb05574a3d9c2a94da84e571a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:58.825546 26019 raft_consensus.cc:1275] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Refusing update from remote peer a68fefdb05574a3d9c2a94da84e571a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:57:58.826414 26379 consensus_queue.cc:1048] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:58.827427 26434 consensus_queue.cc:1048] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:57:58.870616 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.861264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:46039 (local address 127.23.132.66:54569)
0320 02:57:58.861530 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0320 02:57:58.861545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:57:58.861633 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:57:58.862697 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:57:58.862701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:57:58.862711 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:57:58.863037 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.863044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.864989 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.864993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:57:58.867372 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:57:58.867402 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.867852 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.867890 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.870104 (+ 2214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.870113 (+ 9us) client_negotiation.cc:769] Sending connection context
0320 02:57:58.870382 (+ 269us) client_negotiation.cc:240] Negotiation successful
0320 02:57:58.870483 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260320 02:57:58.871423 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:57:58.862299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:54569 (local address 127.23.132.67:46039)
0320 02:57:58.862444 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0320 02:57:58.862449 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:57:58.862465 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:57:58.862515 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:57:58.862519 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:57:58.862577 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:57:58.863157 (+ 580us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:57:58.863255 (+ 98us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.864519 (+ 1264us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:57:58.868512 (+ 3993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:57:58.869616 (+ 1104us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:57:58.869758 (+ 142us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:57:58.869903 (+ 145us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:57:58.871103 (+ 1200us) server_negotiation.cc:1036] Waiting for connection context
0320 02:57:58.871143 (+ 40us) server_negotiation.cc:300] Negotiation successful
0320 02:57:58.871207 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260320 02:57:58.873281 26405 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260320 02:58:00.495313 25860 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:00.495501 25860 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:00.498801 25860 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Sending DeleteTablet for 3 replicas of tablet e5d4bf7c97b4464b93ca2210cd718a95
I20260320 02:58:00.499504 26129 tablet_service.cc:1558] Processing DeleteTablet for tablet e5d4bf7c97b4464b93ca2210cd718a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:00 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38854
I20260320 02:58:00.499747 25998 tablet_service.cc:1558] Processing DeleteTablet for tablet e5d4bf7c97b4464b93ca2210cd718a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:00 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:44954
I20260320 02:58:00.500072 26283 tablet_service.cc:1558] Processing DeleteTablet for tablet e5d4bf7c97b4464b93ca2210cd718a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:00 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39102
I20260320 02:58:00.500413 26472 tablet_replica.cc:333] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: stopping tablet replica
I20260320 02:58:00.500543 26472 raft_consensus.cc:2243] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:58:00.500883 26472 raft_consensus.cc:2272] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:00.501134 26470 tablet_replica.cc:333] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: stopping tablet replica
I20260320 02:58:00.501260 26470 raft_consensus.cc:2243] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:00.501382 26470 raft_consensus.cc:2272] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:00.501641 26471 tablet_replica.cc:333] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: stopping tablet replica
I20260320 02:58:00.502090 26471 raft_consensus.cc:2243] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:00.502296 26471 raft_consensus.cc:2272] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:00.502563 26472 ts_tablet_manager.cc:1916] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:00.502930 26471 ts_tablet_manager.cc:1916] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:00.502923 26470 ts_tablet_manager.cc:1916] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:00.508662 26472 ts_tablet_manager.cc:1929] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260320 02:58:00.508927 26472 log.cc:1199] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/e5d4bf7c97b4464b93ca2210cd718a95
I20260320 02:58:00.509621 26472 ts_tablet_manager.cc:1950] T e5d4bf7c97b4464b93ca2210cd718a95 P a68fefdb05574a3d9c2a94da84e571a0: Deleting consensus metadata
I20260320 02:58:00.511739 26471 ts_tablet_manager.cc:1929] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260320 02:58:00.511700 26470 ts_tablet_manager.cc:1929] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260320 02:58:00.511839 26470 log.cc:1199] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/e5d4bf7c97b4464b93ca2210cd718a95
I20260320 02:58:00.511962 26471 log.cc:1199] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/e5d4bf7c97b4464b93ca2210cd718a95
I20260320 02:58:00.512295 26470 ts_tablet_manager.cc:1950] T e5d4bf7c97b4464b93ca2210cd718a95 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting consensus metadata
I20260320 02:58:00.512295 26471 ts_tablet_manager.cc:1950] T e5d4bf7c97b4464b93ca2210cd718a95 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting consensus metadata
I20260320 02:58:00.513940 25846 catalog_manager.cc:5002] TS 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221): tablet e5d4bf7c97b4464b93ca2210cd718a95 (table test-table [id=597afe4f5fbc44f1aa3e9b7ff5556c46]) successfully deleted
I20260320 02:58:00.513940 25845 catalog_manager.cc:5002] TS 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953): tablet e5d4bf7c97b4464b93ca2210cd718a95 (table test-table [id=597afe4f5fbc44f1aa3e9b7ff5556c46]) successfully deleted
I20260320 02:58:00.514290 25845 catalog_manager.cc:5002] TS a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039): tablet e5d4bf7c97b4464b93ca2210cd718a95 (table test-table [id=597afe4f5fbc44f1aa3e9b7ff5556c46]) successfully deleted
Mar 20 02:58:00 dist-test-slave-v3hj krb5kdc[25811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975480, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 20 02:58:00 dist-test-slave-v3hj krb5kdc[25811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975480, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:00.549369 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:00.535343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.23.132.126:39733)
0320 02:58:00.535620 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:00.535624 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:00.535882 (+ 258us) server_negotiation.cc:408] Connection header received
0320 02:58:00.536150 (+ 268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:00.536154 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:00.536206 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:00.536292 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:00.537205 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:00.537743 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:00.538474 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:00.538637 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:00.541282 (+ 2645us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:00.541306 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:00.541309 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:00.541341 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:00.543514 (+ 2173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:00.546165 (+ 2651us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:00.546173 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:00.546175 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:00.546253 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:00.546839 (+ 586us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:00.546843 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:00.546844 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:00.547343 (+ 499us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:00.547537 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:00.549021 (+ 1484us) server_negotiation.cc:300] Negotiation successful
0320 02:58:00.549196 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260320 02:58:00.557466 25860 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:58:00.558012 25860 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:58:00.564847 25998 tablet_service.cc:1511] Processing CreateTablet for tablet 170bd4240c4a41b9b423f48dccb00b07 (DEFAULT_TABLE table=test-table [id=42b55afca98748248149f619a111e324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:00.565204 25998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 170bd4240c4a41b9b423f48dccb00b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:00.565207 26129 tablet_service.cc:1511] Processing CreateTablet for tablet 170bd4240c4a41b9b423f48dccb00b07 (DEFAULT_TABLE table=test-table [id=42b55afca98748248149f619a111e324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:00.565513 26129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 170bd4240c4a41b9b423f48dccb00b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:00.565542 26283 tablet_service.cc:1511] Processing CreateTablet for tablet 170bd4240c4a41b9b423f48dccb00b07 (DEFAULT_TABLE table=test-table [id=42b55afca98748248149f619a111e324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:00.565811 26283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 170bd4240c4a41b9b423f48dccb00b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:00.570775 26482 tablet_bootstrap.cc:492] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Bootstrap starting.
I20260320 02:58:00.571630 26481 tablet_bootstrap.cc:492] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Bootstrap starting.
I20260320 02:58:00.571578 26483 tablet_bootstrap.cc:492] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Bootstrap starting.
I20260320 02:58:00.572577 26481 tablet_bootstrap.cc:654] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:00.573369 26482 tablet_bootstrap.cc:654] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:00.574795 26483 tablet_bootstrap.cc:654] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:00.574937 26482 tablet_bootstrap.cc:492] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: No bootstrap required, opened a new log
I20260320 02:58:00.575148 26481 tablet_bootstrap.cc:492] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: No bootstrap required, opened a new log
I20260320 02:58:00.575227 26481 ts_tablet_manager.cc:1403] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260320 02:58:00.575227 26482 ts_tablet_manager.cc:1403] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260320 02:58:00.575790 26481 raft_consensus.cc:359] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.575918 26481 raft_consensus.cc:385] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:00.575958 26481 raft_consensus.cc:740] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Initialized, Role: FOLLOWER
I20260320 02:58:00.575941 26482 raft_consensus.cc:359] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.576063 26482 raft_consensus.cc:385] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:00.576097 26482 raft_consensus.cc:740] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68fefdb05574a3d9c2a94da84e571a0, State: Initialized, Role: FOLLOWER
I20260320 02:58:00.576200 26481 consensus_queue.cc:260] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.576232 26482 consensus_queue.cc:260] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.576383 26483 tablet_bootstrap.cc:492] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: No bootstrap required, opened a new log
I20260320 02:58:00.576530 26482 ts_tablet_manager.cc:1434] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:58:00.576699 26481 ts_tablet_manager.cc:1434] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260320 02:58:00.576464 26483 ts_tablet_manager.cc:1403] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260320 02:58:00.577373 26483 raft_consensus.cc:359] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.577468 26483 raft_consensus.cc:385] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:00.577497 26483 raft_consensus.cc:740] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Initialized, Role: FOLLOWER
I20260320 02:58:00.577646 26483 consensus_queue.cc:260] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.578570 26483 ts_tablet_manager.cc:1434] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260320 02:58:00.679376 26487 raft_consensus.cc:493] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:00.679528 26487 raft_consensus.cc:515] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.679956 26487 leader_election.cc:290] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221), a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:58:00.680488 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "170bd4240c4a41b9b423f48dccb00b07" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384" is_pre_election: true
I20260320 02:58:00.680495 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "170bd4240c4a41b9b423f48dccb00b07" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0" is_pre_election: true
I20260320 02:58:00.680621 26159 raft_consensus.cc:2468] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 0.
I20260320 02:58:00.680644 26303 raft_consensus.cc:2468] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 0.
I20260320 02:58:00.681145 25952 leader_election.cc:304] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, 5cca94d7d46d436fb34a2c52e08c6384; no voters:
I20260320 02:58:00.681344 26487 raft_consensus.cc:2804] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:58:00.681399 26487 raft_consensus.cc:493] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:00.681427 26487 raft_consensus.cc:3060] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:00.682425 26487 raft_consensus.cc:515] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.682786 26487 leader_election.cc:290] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 election: Requested vote from peers 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221), a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:58:00.683310 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "170bd4240c4a41b9b423f48dccb00b07" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0"
I20260320 02:58:00.683393 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "170bd4240c4a41b9b423f48dccb00b07" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384"
I20260320 02:58:00.683558 26303 raft_consensus.cc:3060] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:00.683558 26159 raft_consensus.cc:3060] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:00.684719 26303 raft_consensus.cc:2468] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 1.
I20260320 02:58:00.684719 26159 raft_consensus.cc:2468] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 1.
I20260320 02:58:00.685171 25952 leader_election.cc:304] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, 5cca94d7d46d436fb34a2c52e08c6384; no voters:
I20260320 02:58:00.685832 26487 raft_consensus.cc:2804] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:00.686033 26487 raft_consensus.cc:697] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Running, Role: LEADER
I20260320 02:58:00.686149 26487 consensus_queue.cc:237] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.692539 25860 catalog_manager.cc:5671] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65). New cstate: current_term: 1 leader_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } health_report { overall_health: UNKNOWN } } }
I20260320 02:58:00.718662 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:00.715048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.23.132.65:38953)
0320 02:58:00.715348 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:00.715352 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:00.715364 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:00.715412 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:00.715414 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:00.715462 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:00.715539 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:58:00.715974 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:00.716514 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:00.717751 (+ 1237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:00.717979 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:00.718032 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:58:00.718125 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:58:00.718200 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:00.718361 (+ 161us) server_negotiation.cc:300] Negotiation successful
0320 02:58:00.718413 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
I20260320 02:58:00.730446 26159 raft_consensus.cc:1275] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Refusing update from remote peer 1d8e3ba92c9543e0a0eba841e05ddf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:00.730861 26303 raft_consensus.cc:1275] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Refusing update from remote peer 1d8e3ba92c9543e0a0eba841e05ddf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:00.731184 26488 consensus_queue.cc:1048] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:00.731596 26487 consensus_queue.cc:1048] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:00.792171 25858 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:00.793553 25858 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:00.796799 25858 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Sending DeleteTablet for 3 replicas of tablet 170bd4240c4a41b9b423f48dccb00b07
I20260320 02:58:00.797582 25998 tablet_service.cc:1558] Processing DeleteTablet for tablet 170bd4240c4a41b9b423f48dccb00b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:00 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:44954
I20260320 02:58:00.797993 26129 tablet_service.cc:1558] Processing DeleteTablet for tablet 170bd4240c4a41b9b423f48dccb00b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:00 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38854
I20260320 02:58:00.798180 26470 tablet_replica.cc:333] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: stopping tablet replica
I20260320 02:58:00.798286 26470 raft_consensus.cc:2243] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:00.798287 26471 tablet_replica.cc:333] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: stopping tablet replica
I20260320 02:58:00.798399 26470 raft_consensus.cc:2272] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:00.798573 26283 tablet_service.cc:1558] Processing DeleteTablet for tablet 170bd4240c4a41b9b423f48dccb00b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:00 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39102
I20260320 02:58:00.798743 26472 tablet_replica.cc:333] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: stopping tablet replica
I20260320 02:58:00.798832 26472 raft_consensus.cc:2243] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:00.798941 26472 raft_consensus.cc:2272] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:00.798954 26471 raft_consensus.cc:2243] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:58:00.799340 26471 raft_consensus.cc:2272] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:00.799875 26470 ts_tablet_manager.cc:1916] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:00.800252 26472 ts_tablet_manager.cc:1916] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:00.800585 26471 ts_tablet_manager.cc:1916] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:00.806548 26472 ts_tablet_manager.cc:1929] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:58:00.806836 26472 log.cc:1199] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/170bd4240c4a41b9b423f48dccb00b07
I20260320 02:58:00.806630 25858 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260320 02:58:00.807114 26472 ts_tablet_manager.cc:1950] T 170bd4240c4a41b9b423f48dccb00b07 P a68fefdb05574a3d9c2a94da84e571a0: Deleting consensus metadata
W20260320 02:58:00.807466 25858 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:58:00.808728 26471 ts_tablet_manager.cc:1929] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:58:00.808816 26471 log.cc:1199] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/170bd4240c4a41b9b423f48dccb00b07
I20260320 02:58:00.809059 26471 ts_tablet_manager.cc:1950] T 170bd4240c4a41b9b423f48dccb00b07 P 1d8e3ba92c9543e0a0eba841e05ddf90: Deleting consensus metadata
I20260320 02:58:00.809455 26470 ts_tablet_manager.cc:1929] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:58:00.809532 26470 log.cc:1199] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/170bd4240c4a41b9b423f48dccb00b07
I20260320 02:58:00.810271 26470 ts_tablet_manager.cc:1950] T 170bd4240c4a41b9b423f48dccb00b07 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting consensus metadata
I20260320 02:58:00.813468 25846 catalog_manager.cc:5002] TS 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221): tablet 170bd4240c4a41b9b423f48dccb00b07 (table test-table [id=42b55afca98748248149f619a111e324]) successfully deleted
I20260320 02:58:00.814136 25845 catalog_manager.cc:5002] TS a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039): tablet 170bd4240c4a41b9b423f48dccb00b07 (table test-table [id=42b55afca98748248149f619a111e324]) successfully deleted
I20260320 02:58:00.814558 25845 catalog_manager.cc:5002] TS 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953): tablet 170bd4240c4a41b9b423f48dccb00b07 (table test-table [id=42b55afca98748248149f619a111e324]) successfully deleted
I20260320 02:58:00.817901 25998 tablet_service.cc:1511] Processing CreateTablet for tablet a3cba7ca57c8435787441ff3dd376f13 (DEFAULT_TABLE table=test-table [id=749915413311496192411abbb54e0072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:00.817996 26129 tablet_service.cc:1511] Processing CreateTablet for tablet a3cba7ca57c8435787441ff3dd376f13 (DEFAULT_TABLE table=test-table [id=749915413311496192411abbb54e0072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:00.818272 26129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cba7ca57c8435787441ff3dd376f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:00.818277 25998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cba7ca57c8435787441ff3dd376f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:00.820695 26481 tablet_bootstrap.cc:492] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Bootstrap starting.
I20260320 02:58:00.822337 26481 tablet_bootstrap.cc:654] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:00.823009 26483 tablet_bootstrap.cc:492] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90: Bootstrap starting.
I20260320 02:58:00.822958 26283 tablet_service.cc:1511] Processing CreateTablet for tablet a3cba7ca57c8435787441ff3dd376f13 (DEFAULT_TABLE table=test-table [id=749915413311496192411abbb54e0072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:00.823246 26283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cba7ca57c8435787441ff3dd376f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:00.824086 26481 tablet_bootstrap.cc:492] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: No bootstrap required, opened a new log
I20260320 02:58:00.824537 26483 tablet_bootstrap.cc:654] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:00.824611 26481 ts_tablet_manager.cc:1403] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:00.825472 26481 raft_consensus.cc:359] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.825589 26481 raft_consensus.cc:385] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:00.825636 26481 raft_consensus.cc:740] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cca94d7d46d436fb34a2c52e08c6384, State: Initialized, Role: FOLLOWER
I20260320 02:58:00.826697 26481 consensus_queue.cc:260] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.827569 26481 ts_tablet_manager.cc:1434] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260320 02:58:00.829967 26482 tablet_bootstrap.cc:492] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Bootstrap starting.
I20260320 02:58:00.833145 26482 tablet_bootstrap.cc:654] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:00.833463 26483 tablet_bootstrap.cc:492] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90: No bootstrap required, opened a new log
I20260320 02:58:00.833578 26483 ts_tablet_manager.cc:1403] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260320 02:58:00.834110 26483 raft_consensus.cc:359] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.834426 26483 raft_consensus.cc:385] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:00.834472 26483 raft_consensus.cc:740] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Initialized, Role: FOLLOWER
I20260320 02:58:00.834755 26483 consensus_queue.cc:260] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.834862 26482 tablet_bootstrap.cc:492] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: No bootstrap required, opened a new log
I20260320 02:58:00.834952 26482 ts_tablet_manager.cc:1403] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260320 02:58:00.835049 26483 ts_tablet_manager.cc:1434] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260320 02:58:00.835448 26482 raft_consensus.cc:359] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.835566 26482 raft_consensus.cc:385] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:00.835598 26482 raft_consensus.cc:740] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68fefdb05574a3d9c2a94da84e571a0, State: Initialized, Role: FOLLOWER
I20260320 02:58:00.836781 26482 consensus_queue.cc:260] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.838375 26482 ts_tablet_manager.cc:1434] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:00.856913 26488 raft_consensus.cc:493] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:00.857059 26488 raft_consensus.cc:515] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.857456 26488 leader_election.cc:290] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221), a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:58:00.858036 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cba7ca57c8435787441ff3dd376f13" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0" is_pre_election: true
I20260320 02:58:00.858036 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cba7ca57c8435787441ff3dd376f13" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384" is_pre_election: true
I20260320 02:58:00.858180 26159 raft_consensus.cc:2468] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 0.
I20260320 02:58:00.858186 26303 raft_consensus.cc:2468] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 0.
I20260320 02:58:00.858513 25952 leader_election.cc:304] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, 5cca94d7d46d436fb34a2c52e08c6384; no voters:
I20260320 02:58:00.858834 26488 raft_consensus.cc:2804] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:58:00.858901 26488 raft_consensus.cc:493] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:00.858930 26488 raft_consensus.cc:3060] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:00.859916 26488 raft_consensus.cc:515] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.860306 26488 leader_election.cc:290] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 election: Requested vote from peers 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221), a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039)
I20260320 02:58:00.861042 26303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cba7ca57c8435787441ff3dd376f13" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68fefdb05574a3d9c2a94da84e571a0"
I20260320 02:58:00.861289 26303 raft_consensus.cc:3060] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:00.861325 26159 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cba7ca57c8435787441ff3dd376f13" candidate_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca94d7d46d436fb34a2c52e08c6384"
I20260320 02:58:00.861434 26159 raft_consensus.cc:3060] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:00.862360 26303 raft_consensus.cc:2468] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 1.
I20260320 02:58:00.862401 26159 raft_consensus.cc:2468] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d8e3ba92c9543e0a0eba841e05ddf90 in term 1.
I20260320 02:58:00.862723 25952 leader_election.cc:304] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d8e3ba92c9543e0a0eba841e05ddf90, 5cca94d7d46d436fb34a2c52e08c6384; no voters:
I20260320 02:58:00.862941 26488 raft_consensus.cc:2804] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:00.863178 26488 raft_consensus.cc:697] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8e3ba92c9543e0a0eba841e05ddf90, State: Running, Role: LEADER
I20260320 02:58:00.863329 26488 consensus_queue.cc:237] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } }
I20260320 02:58:00.866811 25857 catalog_manager.cc:5671] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65). New cstate: current_term: 1 leader_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d8e3ba92c9543e0a0eba841e05ddf90" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 } health_report { overall_health: UNKNOWN } } }
I20260320 02:58:00.903796 26159 raft_consensus.cc:1275] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Refusing update from remote peer 1d8e3ba92c9543e0a0eba841e05ddf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:00.903904 26303 raft_consensus.cc:1275] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Refusing update from remote peer 1d8e3ba92c9543e0a0eba841e05ddf90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:00.904452 26487 consensus_queue.cc:1048] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cca94d7d46d436fb34a2c52e08c6384" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:00.905318 26488 consensus_queue.cc:1048] T a3cba7ca57c8435787441ff3dd376f13 P 1d8e3ba92c9543e0a0eba841e05ddf90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68fefdb05574a3d9c2a94da84e571a0" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260320 02:58:00.930748 26452 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260320 02:58:02.516160 25860 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:02.516331 25860 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:02.519019 25860 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 40d76999fd4b46b0893906374db6a6ef: Sending DeleteTablet for 3 replicas of tablet a3cba7ca57c8435787441ff3dd376f13
I20260320 02:58:02.519729 26283 tablet_service.cc:1558] Processing DeleteTablet for tablet a3cba7ca57c8435787441ff3dd376f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:02 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:39102
I20260320 02:58:02.519635 25998 tablet_service.cc:1558] Processing DeleteTablet for tablet a3cba7ca57c8435787441ff3dd376f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:02 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:44954
I20260320 02:58:02.520057 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25934
I20260320 02:58:02.520144 26524 tablet_replica.cc:333] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: stopping tablet replica
I20260320 02:58:02.521147 26129 tablet_service.cc:1558] Processing DeleteTablet for tablet a3cba7ca57c8435787441ff3dd376f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:02 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38854
I20260320 02:58:02.521313 26524 raft_consensus.cc:2243] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:02.521458 26524 raft_consensus.cc:2272] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:02.521618 26526 tablet_replica.cc:333] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: stopping tablet replica
I20260320 02:58:02.521744 26526 raft_consensus.cc:2243] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:02.522099 26526 raft_consensus.cc:2272] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:02.522354 26524 ts_tablet_manager.cc:1916] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:02.522842 26526 ts_tablet_manager.cc:1916] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:02.525221 26524 ts_tablet_manager.cc:1929] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260320 02:58:02.525309 26524 log.cc:1199] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/a3cba7ca57c8435787441ff3dd376f13
I20260320 02:58:02.525576 26526 ts_tablet_manager.cc:1929] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260320 02:58:02.525604 26524 ts_tablet_manager.cc:1950] T a3cba7ca57c8435787441ff3dd376f13 P a68fefdb05574a3d9c2a94da84e571a0: Deleting consensus metadata
I20260320 02:58:02.525650 26526 log.cc:1199] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/a3cba7ca57c8435787441ff3dd376f13
I20260320 02:58:02.526083 26526 ts_tablet_manager.cc:1950] T a3cba7ca57c8435787441ff3dd376f13 P 5cca94d7d46d436fb34a2c52e08c6384: Deleting consensus metadata
I20260320 02:58:02.527488 25845 catalog_manager.cc:5002] TS a68fefdb05574a3d9c2a94da84e571a0 (127.23.132.67:46039): tablet a3cba7ca57c8435787441ff3dd376f13 (table test-table [id=749915413311496192411abbb54e0072]) successfully deleted
I20260320 02:58:02.527495 25846 catalog_manager.cc:5002] TS 5cca94d7d46d436fb34a2c52e08c6384 (127.23.132.66:44221): tablet a3cba7ca57c8435787441ff3dd376f13 (table test-table [id=749915413311496192411abbb54e0072]) successfully deleted
I20260320 02:58:02.534929 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 26076
W20260320 02:58:02.535066 25845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.132.65:38953 (error 108)
I20260320 02:58:02.538188 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:02.537323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38953 (local address 127.0.0.1:57444)
0320 02:58:02.537717 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:02.537847 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.65:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":160,"threads_started":1}
W20260320 02:58:02.538394 25845 catalog_manager.cc:4729] TS 1d8e3ba92c9543e0a0eba841e05ddf90 (127.23.132.65:38953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3cba7ca57c8435787441ff3dd376f13: Network error: Client connection negotiation failed: client connection to 127.23.132.65:38953: connect: Connection refused (error 111)
I20260320 02:58:02.548466 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 26218
I20260320 02:58:02.557936 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 25828
2026-03-20T02:58:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8464 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:02 dist-test-slave-v3hj krb5kdc[26531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:02 dist-test-slave-v3hj krb5kdc[26531](info): set up 2 sockets
Mar 20 02:58:02 dist-test-slave-v3hj krb5kdc[26531](info): commencing operation
krb5kdc: starting...
W20260320 02:58:04.878744 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.297s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:04 dist-test-slave-v3hj krb5kdc[26531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:05.082187 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:46107
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:46107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:05.232640 26547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:05.233839 26547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:05.234020 26547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:05.239078 26547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:05.239178 26547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:05.239214 26547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:05.239240 26547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:05.239270 26547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:05.244824 26547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:46107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:46107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:05.248445 26547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:05.251232 26547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:05.258823 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:05.258823 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:05.259140 26555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:05.259449 26547 server_base.cc:1061] running on GCE node
I20260320 02:58:05.260397 26547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:05.261408 26547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:05.262692 26547 hybrid_clock.cc:648] HybridClock initialized: now 1773975485262660 us; error 47 us; skew 500 ppm
Mar 20 02:58:05 dist-test-slave-v3hj krb5kdc[26531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:05.265931 26547 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:05.267377 26547 webserver.cc:492] Webserver started at http://127.23.132.126:38721/ using document root <none> and password file <none>
I20260320 02:58:05.267939 26547 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:05.267994 26547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:05.268181 26547 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:05.271641 26547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "aad39631b1ef426c83eab7b49c6d2b4f"
format_stamp: "Formatted at 2026-03-20 02:58:05 on dist-test-slave-v3hj"
server_key: "1f973d7e3a7aad63dc409d7401a8870c"
server_key_iv: "2720f7a003d017dd47faa179fc9091a4"
server_key_version: "encryptionkey@0"
I20260320 02:58:05.272181 26547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "aad39631b1ef426c83eab7b49c6d2b4f"
format_stamp: "Formatted at 2026-03-20 02:58:05 on dist-test-slave-v3hj"
server_key: "1f973d7e3a7aad63dc409d7401a8870c"
server_key_iv: "2720f7a003d017dd47faa179fc9091a4"
server_key_version: "encryptionkey@0"
I20260320 02:58:05.276820 26547 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:58:05.280383 26562 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:05.281656 26547 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:05.281813 26547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "aad39631b1ef426c83eab7b49c6d2b4f"
format_stamp: "Formatted at 2026-03-20 02:58:05 on dist-test-slave-v3hj"
server_key: "1f973d7e3a7aad63dc409d7401a8870c"
server_key_iv: "2720f7a003d017dd47faa179fc9091a4"
server_key_version: "encryptionkey@0"
I20260320 02:58:05.281950 26547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:05.314103 26547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:05.323266 26547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:05.323482 26547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:05.339046 26547 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:46107
I20260320 02:58:05.339109 26615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:46107 every 8 connection(s)
I20260320 02:58:05.340564 26547 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:05.342063 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 26547
I20260320 02:58:05.342177 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:05.342381 24081 external_mini_cluster.cc:1468] Setting key 35bd175410508749f66ab75e2b82ad26
I20260320 02:58:05.345263 26616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:05.352490 26616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f: Bootstrap starting.
Mar 20 02:58:05 dist-test-slave-v3hj krb5kdc[26531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:05.356312 26616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:05.357158 26616 log.cc:826] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:05.359763 26616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f: No bootstrap required, opened a new log
I20260320 02:58:05.362663 26616 raft_consensus.cc:359] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46107 } }
I20260320 02:58:05.362960 26616 raft_consensus.cc:385] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:05.363029 26616 raft_consensus.cc:740] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad39631b1ef426c83eab7b49c6d2b4f, State: Initialized, Role: FOLLOWER
I20260320 02:58:05.363032 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:05.344174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.23.132.126:46107)
0320 02:58:05.344962 (+ 788us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:05.344972 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:05.345009 (+ 37us) server_negotiation.cc:408] Connection header received
0320 02:58:05.347288 (+ 2279us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:05.347311 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:05.347607 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:05.347910 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:05.350612 (+ 2702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:05.351713 (+ 1101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:05.353015 (+ 1302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:05.353312 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:05.356539 (+ 3227us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:05.356560 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:05.356573 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:05.356622 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:05.360292 (+ 3670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:05.360947 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:05.360953 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:05.360960 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:05.361049 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:05.361278 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:05.361281 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:05.361283 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:05.361637 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:05.361858 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:05.362145 (+ 287us) server_negotiation.cc:300] Negotiation successful
0320 02:58:05.362404 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":217,"threads_started":1}
I20260320 02:58:05.363641 26616 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46107 } }
I20260320 02:58:05.364781 26616 raft_consensus.cc:399] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:05.364856 26616 raft_consensus.cc:493] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:05.364971 26616 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:05.368878 26616 raft_consensus.cc:515] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46107 } }
I20260320 02:58:05.369287 26616 leader_election.cc:304] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad39631b1ef426c83eab7b49c6d2b4f; no voters:
I20260320 02:58:05.369618 26616 leader_election.cc:290] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:05.370463 26621 raft_consensus.cc:2804] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:05.371795 26621 raft_consensus.cc:697] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [term 1 LEADER]: Becoming Leader. State: Replica: aad39631b1ef426c83eab7b49c6d2b4f, State: Running, Role: LEADER
I20260320 02:58:05.374207 26621 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46107 } }
I20260320 02:58:05.374485 26616 sys_catalog.cc:565] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:05.377213 26623 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad39631b1ef426c83eab7b49c6d2b4f. Latest consensus state: current_term: 1 leader_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46107 } } }
I20260320 02:58:05.377355 26623 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:05.377804 26630 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:05.377686 26622 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad39631b1ef426c83eab7b49c6d2b4f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46107 } } }
I20260320 02:58:05.377874 26622 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:05.380887 26630 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:05.391458 26630 catalog_manager.cc:1357] Generated new cluster ID: 1b054c2815d54e3c98120e626283fb85
I20260320 02:58:05.391574 26630 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:05.431700 26630 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:05.432889 26630 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:05.454478 26630 catalog_manager.cc:6044] T 00000000000000000000000000000000 P aad39631b1ef426c83eab7b49c6d2b4f: Generated new TSK 0
I20260320 02:58:05.455577 26630 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:05.570129 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46107
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:05.748345 26644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:05.748795 26644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:05.749588 26644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:05.755597 26644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:05.755717 26644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:05.755816 26644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:05.760892 26644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:05.762245 26644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:05.763367 26644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:05.775019 26650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:05.775259 26644 server_base.cc:1061] running on GCE node
W20260320 02:58:05.775279 26649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:05.775279 26652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:05.776434 26644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:05.777158 26644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:05.780496 26644 hybrid_clock.cc:648] HybridClock initialized: now 1773975485780370 us; error 140 us; skew 500 ppm
Mar 20 02:58:05 dist-test-slave-v3hj krb5kdc[26531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:05.783627 26644 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:58:05.785151 26644 webserver.cc:492] Webserver started at http://127.23.132.65:36213/ using document root <none> and password file <none>
I20260320 02:58:05.785712 26644 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:05.785777 26644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:05.786092 26644 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:05.790257 26644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "7b11c3b026204b1b8217b91cc42b9a6b"
format_stamp: "Formatted at 2026-03-20 02:58:05 on dist-test-slave-v3hj"
server_key: "93bafa574a9411f6c25ba11360a7ceac"
server_key_iv: "373c2bbb371c7e3ab1c2d53686f6baa2"
server_key_version: "encryptionkey@0"
I20260320 02:58:05.790958 26644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "7b11c3b026204b1b8217b91cc42b9a6b"
format_stamp: "Formatted at 2026-03-20 02:58:05 on dist-test-slave-v3hj"
server_key: "93bafa574a9411f6c25ba11360a7ceac"
server_key_iv: "373c2bbb371c7e3ab1c2d53686f6baa2"
server_key_version: "encryptionkey@0"
I20260320 02:58:05.797391 26644 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260320 02:58:05.799925 26659 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:05.801717 26644 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260320 02:58:05.802219 26644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "7b11c3b026204b1b8217b91cc42b9a6b"
format_stamp: "Formatted at 2026-03-20 02:58:05 on dist-test-slave-v3hj"
server_key: "93bafa574a9411f6c25ba11360a7ceac"
server_key_iv: "373c2bbb371c7e3ab1c2d53686f6baa2"
server_key_version: "encryptionkey@0"
I20260320 02:58:05.802361 26644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:05.821596 26644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:05.830845 26644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:05.831089 26644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:05.832854 26644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:05.834127 26644 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:05.834192 26644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:05.834246 26644 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:05.834275 26644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:05.852581 26644 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:46179
I20260320 02:58:05.852599 26772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:46179 every 8 connection(s)
I20260320 02:58:05.853863 26644 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:05.862464 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 26644
I20260320 02:58:05.862596 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:05.862839 24081 external_mini_cluster.cc:1468] Setting key b990d07d60be3bdce8718b394a8de486
Mar 20 02:58:05 dist-test-slave-v3hj krb5kdc[26531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:05.875939 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:05.856018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:44667 (local address 127.23.132.126:46107)
0320 02:58:05.856148 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:05.856151 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:05.857158 (+ 1007us) server_negotiation.cc:408] Connection header received
0320 02:58:05.858083 (+ 925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:05.858086 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:05.858139 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:05.858215 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:05.861747 (+ 3532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:05.862362 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:05.863482 (+ 1120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:05.863741 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:05.868555 (+ 4814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:05.868582 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:05.868585 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:05.868618 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:05.871195 (+ 2577us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:05.873141 (+ 1946us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:05.873152 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:05.873154 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:05.873387 (+ 233us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:05.874410 (+ 1023us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:05.874415 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:05.874416 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:05.874641 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:05.874762 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:05.875634 (+ 872us) server_negotiation.cc:300] Negotiation successful
0320 02:58:05.875796 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260320 02:58:05.877054 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:05.856166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46107 (local address 127.23.132.65:44667)
0320 02:58:05.856822 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:05.857018 (+ 196us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:05.857872 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:05.859170 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:05.859185 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:05.859792 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:05.861501 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:05.861525 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:05.862530 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:05.862535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:05.863247 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:05.863261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:05.863500 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:05.864262 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:05.864300 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:05.868295 (+ 3995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:05.871482 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:05.871495 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:05.871515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:05.872387 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:05.873642 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:05.873652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:05.873654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:05.874201 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:05.874783 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:05.874791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:05.875277 (+ 486us) client_negotiation.cc:769] Sending connection context
0320 02:58:05.875542 (+ 265us) client_negotiation.cc:240] Negotiation successful
0320 02:58:05.875901 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:05.879120 26773 heartbeater.cc:344] Connected to a master server at 127.23.132.126:46107
I20260320 02:58:05.879393 26773 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:05.880014 26773 heartbeater.cc:507] Master 127.23.132.126:46107 requested a full tablet report, sending...
I20260320 02:58:05.881600 26580 ts_manager.cc:194] Registered new tserver with Master: 7b11c3b026204b1b8217b91cc42b9a6b (127.23.132.65:46179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:58:05.882822 26580 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:44667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:05.959321 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46107
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:06.121438 26780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:06.121767 26780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:06.122046 26780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:06.125902 26780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:06.125991 26780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:06.126103 26780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:06.133766 26780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:06.135501 26780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:06.136444 26780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:06.146575 26788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:06.147353 26780 server_base.cc:1061] running on GCE node
W20260320 02:58:06.147778 26786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:06.148413 26785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:06.149602 26780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:06.151213 26780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:06.154894 26780 hybrid_clock.cc:648] HybridClock initialized: now 1773975486154829 us; error 83 us; skew 500 ppm
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[26531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:06.158633 26780 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:58:06.160423 26780 webserver.cc:492] Webserver started at http://127.23.132.66:38971/ using document root <none> and password file <none>
I20260320 02:58:06.161006 26780 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:06.161059 26780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:06.161247 26780 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:06.163398 26780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "35639ea74a4c4289a107437679c0e5b7"
format_stamp: "Formatted at 2026-03-20 02:58:06 on dist-test-slave-v3hj"
server_key: "0795ca009baef4f2f96e6c724a6793a9"
server_key_iv: "c419531cc5cf28e6b60b50fe312129e0"
server_key_version: "encryptionkey@0"
I20260320 02:58:06.164287 26780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "35639ea74a4c4289a107437679c0e5b7"
format_stamp: "Formatted at 2026-03-20 02:58:06 on dist-test-slave-v3hj"
server_key: "0795ca009baef4f2f96e6c724a6793a9"
server_key_iv: "c419531cc5cf28e6b60b50fe312129e0"
server_key_version: "encryptionkey@0"
I20260320 02:58:06.170485 26780 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260320 02:58:06.174332 26795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:06.175741 26780 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260320 02:58:06.175864 26780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "35639ea74a4c4289a107437679c0e5b7"
format_stamp: "Formatted at 2026-03-20 02:58:06 on dist-test-slave-v3hj"
server_key: "0795ca009baef4f2f96e6c724a6793a9"
server_key_iv: "c419531cc5cf28e6b60b50fe312129e0"
server_key_version: "encryptionkey@0"
I20260320 02:58:06.175976 26780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:06.234820 26780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:06.241660 26780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:06.242202 26780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:06.242951 26780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:06.244326 26780 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:06.244424 26780 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:06.244495 26780 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:06.244841 26780 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:06.263141 26780 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:34875
I20260320 02:58:06.263208 26908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:34875 every 8 connection(s)
I20260320 02:58:06.264161 26780 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:06.269975 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 26780
I20260320 02:58:06.270131 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:06.270385 24081 external_mini_cluster.cc:1468] Setting key 2dbfe02ab184ded8d3444658604db983
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[26531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:58:06.284086 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:06.267322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:33501 (local address 127.23.132.126:46107)
0320 02:58:06.267485 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:06.267488 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:06.270113 (+ 2625us) server_negotiation.cc:408] Connection header received
0320 02:58:06.271374 (+ 1261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:06.271378 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:06.271445 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:06.271523 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:06.274916 (+ 3393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.275419 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:06.276392 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.276591 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:06.279923 (+ 3332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:06.279953 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:06.279964 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:06.279999 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:06.281549 (+ 1550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:06.282129 (+ 580us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:06.282132 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:06.282133 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:06.282181 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:06.282672 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:06.282675 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:06.282676 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:06.282870 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:06.283132 (+ 262us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:06.283728 (+ 596us) server_negotiation.cc:300] Negotiation successful
0320 02:58:06.283883 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260320 02:58:06.284724 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:06.268011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46107 (local address 127.23.132.66:33501)
0320 02:58:06.269924 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:06.269985 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:06.271134 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:06.271698 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:06.271707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:06.273150 (+ 1443us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:06.274680 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:06.274716 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.275569 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:06.275573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:06.276190 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:06.276202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.276533 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:06.277474 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:06.277504 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:06.279618 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:06.281701 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:06.281708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:06.281718 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:06.282017 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:06.282259 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:06.282262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:06.282264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:06.282537 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:06.283205 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:06.283210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:06.283476 (+ 266us) client_negotiation.cc:769] Sending connection context
0320 02:58:06.283661 (+ 185us) client_negotiation.cc:240] Negotiation successful
0320 02:58:06.283965 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"thread_start_us":676,"threads_started":1}
I20260320 02:58:06.288913 26909 heartbeater.cc:344] Connected to a master server at 127.23.132.126:46107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
I20260320 02:58:06.290498 26909 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:06.291438 26909 heartbeater.cc:507] Master 127.23.132.126:46107 requested a full tablet report, sending...
I20260320 02:58:06.293601 26580 ts_manager.cc:194] Registered new tserver with Master: 35639ea74a4c4289a107437679c0e5b7 (127.23.132.66:34875)
I20260320 02:58:06.294457 26580 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:33501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:06.360658 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46107
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:06.535046 26916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:06.535298 26916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:06.535360 26916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:06.538973 26916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:06.539054 26916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:06.539140 26916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:06.546888 26916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:46107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:06.549046 26916 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:06.551723 26916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:06.561915 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:06.561995 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:06.562870 26921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:06.563172 26916 server_base.cc:1061] running on GCE node
I20260320 02:58:06.563719 26916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:06.565477 26916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:06.567704 26916 hybrid_clock.cc:648] HybridClock initialized: now 1773975486567640 us; error 77 us; skew 500 ppm
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[26531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:06.573393 26916 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:58:06.575162 26916 webserver.cc:492] Webserver started at http://127.23.132.67:36389/ using document root <none> and password file <none>
I20260320 02:58:06.576186 26916 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:06.576246 26916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:06.576462 26916 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:06.578413 26916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "b4be0abc09044be48639086aea57faff"
format_stamp: "Formatted at 2026-03-20 02:58:06 on dist-test-slave-v3hj"
server_key: "64d6670eccded71024010a44094a43ff"
server_key_iv: "c72f2dc21f5f622f120e3ae5c579c3f6"
server_key_version: "encryptionkey@0"
I20260320 02:58:06.578915 26916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "b4be0abc09044be48639086aea57faff"
format_stamp: "Formatted at 2026-03-20 02:58:06 on dist-test-slave-v3hj"
server_key: "64d6670eccded71024010a44094a43ff"
server_key_iv: "c72f2dc21f5f622f120e3ae5c579c3f6"
server_key_version: "encryptionkey@0"
I20260320 02:58:06.583343 26916 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:58:06.590130 26931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:06.594475 26916 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.009s sys 0.001s
I20260320 02:58:06.594635 26916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "b4be0abc09044be48639086aea57faff"
format_stamp: "Formatted at 2026-03-20 02:58:06 on dist-test-slave-v3hj"
server_key: "64d6670eccded71024010a44094a43ff"
server_key_iv: "c72f2dc21f5f622f120e3ae5c579c3f6"
server_key_version: "encryptionkey@0"
I20260320 02:58:06.595026 26916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:06.623476 26916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:06.633828 26916 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:06.634218 26916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:06.634833 26916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:06.635766 26916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:06.635813 26916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:06.635852 26916 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:06.635882 26916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:06.661453 26916 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:44343
I20260320 02:58:06.661497 27044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:44343 every 8 connection(s)
I20260320 02:58:06.662914 26916 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:06.673377 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 26916
I20260320 02:58:06.673749 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:06.674237 24081 external_mini_cluster.cc:1468] Setting key 4efc4d24e6f4fd3a0e2b206e236069d5
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[26531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975486, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:06.685228 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:06.667191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:53407 (local address 127.23.132.126:46107)
0320 02:58:06.667345 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:06.667349 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:06.668263 (+ 914us) server_negotiation.cc:408] Connection header received
0320 02:58:06.669176 (+ 913us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:06.669180 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:06.669242 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:06.669316 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:06.673956 (+ 4640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.674551 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:06.675250 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.675421 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:06.679927 (+ 4506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:06.679948 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:06.679950 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:06.679986 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:06.681725 (+ 1739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:06.682270 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:06.682274 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:06.682275 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:06.682321 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:06.682614 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:06.682617 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:06.682619 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:06.682803 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:06.682890 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:06.683579 (+ 689us) server_negotiation.cc:300] Negotiation successful
0320 02:58:06.683722 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260320 02:58:06.688699 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:06.667479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46107 (local address 127.23.132.67:53407)
0320 02:58:06.668104 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:06.668145 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:06.668984 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:06.669481 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:06.669490 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:06.670496 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:06.671478 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:06.671495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.674711 (+ 3216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:06.674716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:06.675127 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:06.675137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:06.675370 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:06.676831 (+ 1461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:06.676863 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:06.679755 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:06.681882 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:06.681890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:06.681904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:06.682171 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:06.682402 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:06.682406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:06.682408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:06.682526 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:06.682903 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:06.682910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:06.683307 (+ 397us) client_negotiation.cc:769] Sending connection context
0320 02:58:06.683526 (+ 219us) client_negotiation.cc:240] Negotiation successful
0320 02:58:06.685216 (+ 1690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":151,"threads_started":1}
I20260320 02:58:06.691854 27045 heartbeater.cc:344] Connected to a master server at 127.23.132.126:46107
I20260320 02:58:06.692159 27045 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:06.692890 27045 heartbeater.cc:507] Master 127.23.132.126:46107 requested a full tablet report, sending...
I20260320 02:58:06.694185 26579 ts_manager.cc:194] Registered new tserver with Master: b4be0abc09044be48639086aea57faff (127.23.132.67:44343)
I20260320 02:58:06.694808 26579 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:53407
I20260320 02:58:06.705889 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260320 02:58:06.721328 26619 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:06.719590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.23.132.126:46107)
0320 02:58:06.719915 (+ 325us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:06.719920 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:06.720058 (+ 138us) server_negotiation.cc:408] Connection header received
0320 02:58:06.720205 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:06.720209 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:06.720261 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:06.720353 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:06.721197 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59760: BlockingRecv error: recv got EOF from 127.0.0.1:59760 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260320 02:58:06.723423 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 26644
I20260320 02:58:06.729988 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 26780
I20260320 02:58:06.737020 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 26916
I20260320 02:58:06.745854 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 26547
2026-03-20T02:58:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4203 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[27057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[27057](info): set up 2 sockets
Mar 20 02:58:06 dist-test-slave-v3hj krb5kdc[27057](info): commencing operation
krb5kdc: starting...
W20260320 02:58:09.027560 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.239s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:09 dist-test-slave-v3hj krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:09.223402 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:38405
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:09.361850 27073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:09.362103 27073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:09.362159 27073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:09.366901 27073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:09.367316 27073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:09.367516 27073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:09.367548 27073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:09.367589 27073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:09.373714 27073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:38405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:38405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:09.375464 27073 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:09.376654 27073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:09.386749 27081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:09.386871 27079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:09.386888 27078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:09.387867 27073 server_base.cc:1061] running on GCE node
I20260320 02:58:09.388562 27073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:09.390233 27073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:09.391422 27073 hybrid_clock.cc:648] HybridClock initialized: now 1773975489391487 us; error 126 us; skew 500 ppm
Mar 20 02:58:09 dist-test-slave-v3hj krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:09.395381 27073 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:09.396605 27073 webserver.cc:492] Webserver started at http://127.23.132.126:44785/ using document root <none> and password file <none>
I20260320 02:58:09.397181 27073 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:09.397578 27073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:09.398561 27073 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:09.400763 27073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "0975caafd9d44184a56b846d4cd6b31f"
format_stamp: "Formatted at 2026-03-20 02:58:09 on dist-test-slave-v3hj"
server_key: "ec0a53ed101fc0d8e257efc7e61eca64"
server_key_iv: "d6ebd357886280dc5ccc7ebb86fcbaf3"
server_key_version: "encryptionkey@0"
I20260320 02:58:09.401458 27073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "0975caafd9d44184a56b846d4cd6b31f"
format_stamp: "Formatted at 2026-03-20 02:58:09 on dist-test-slave-v3hj"
server_key: "ec0a53ed101fc0d8e257efc7e61eca64"
server_key_iv: "d6ebd357886280dc5ccc7ebb86fcbaf3"
server_key_version: "encryptionkey@0"
I20260320 02:58:09.407223 27073 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260320 02:58:09.411011 27088 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:09.412885 27073 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260320 02:58:09.413102 27073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "0975caafd9d44184a56b846d4cd6b31f"
format_stamp: "Formatted at 2026-03-20 02:58:09 on dist-test-slave-v3hj"
server_key: "ec0a53ed101fc0d8e257efc7e61eca64"
server_key_iv: "d6ebd357886280dc5ccc7ebb86fcbaf3"
server_key_version: "encryptionkey@0"
I20260320 02:58:09.413233 27073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:09.433851 27073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:09.438174 27073 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:09.438354 27073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:09.450333 27073 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:38405
I20260320 02:58:09.450356 27140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:38405 every 8 connection(s)
I20260320 02:58:09.452265 27073 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:09.452728 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27073
I20260320 02:58:09.452838 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:09.453401 24081 external_mini_cluster.cc:1468] Setting key c62079c73a35eaf2c87dc5edcc34e04e
I20260320 02:58:09.457048 27141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:09.467276 27141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f: Bootstrap starting.
Mar 20 02:58:09 dist-test-slave-v3hj krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:09.470538 27141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:09.471398 27141 log.cc:826] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:09.474872 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:09.455188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.23.132.126:38405)
0320 02:58:09.455688 (+ 500us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:09.455699 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:09.455733 (+ 34us) server_negotiation.cc:408] Connection header received
0320 02:58:09.456416 (+ 683us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:09.456432 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:09.456703 (+ 271us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:09.457019 (+ 316us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:09.459125 (+ 2106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:09.461832 (+ 2707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:09.463057 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:09.463369 (+ 312us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:09.468431 (+ 5062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:09.468497 (+ 66us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:09.468510 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:09.468553 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:09.470598 (+ 2045us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:09.471392 (+ 794us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:09.471397 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:09.471402 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:09.471482 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:09.472468 (+ 986us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:09.472472 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:09.472474 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:09.472991 (+ 517us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:09.473164 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:09.473763 (+ 599us) server_negotiation.cc:300] Negotiation successful
0320 02:58:09.474230 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":149,"threads_started":1}
I20260320 02:58:09.475170 27141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f: No bootstrap required, opened a new log
I20260320 02:58:09.478024 27141 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975caafd9d44184a56b846d4cd6b31f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38405 } }
I20260320 02:58:09.478219 27141 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:09.478288 27141 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0975caafd9d44184a56b846d4cd6b31f, State: Initialized, Role: FOLLOWER
I20260320 02:58:09.478996 27141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975caafd9d44184a56b846d4cd6b31f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38405 } }
I20260320 02:58:09.479146 27141 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:09.479212 27141 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:09.479387 27141 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:09.480413 27141 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975caafd9d44184a56b846d4cd6b31f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38405 } }
I20260320 02:58:09.480742 27141 leader_election.cc:304] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0975caafd9d44184a56b846d4cd6b31f; no voters:
I20260320 02:58:09.481372 27141 leader_election.cc:290] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:09.482038 27146 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:09.482252 27146 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [term 1 LEADER]: Becoming Leader. State: Replica: 0975caafd9d44184a56b846d4cd6b31f, State: Running, Role: LEADER
I20260320 02:58:09.482765 27146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975caafd9d44184a56b846d4cd6b31f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38405 } }
I20260320 02:58:09.482918 27141 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:09.484768 27147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0975caafd9d44184a56b846d4cd6b31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975caafd9d44184a56b846d4cd6b31f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38405 } } }
I20260320 02:58:09.484908 27147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:09.485168 27148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0975caafd9d44184a56b846d4cd6b31f. Latest consensus state: current_term: 1 leader_uuid: "0975caafd9d44184a56b846d4cd6b31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0975caafd9d44184a56b846d4cd6b31f" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38405 } } }
I20260320 02:58:09.485239 27148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:09.485482 27155 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:09.494298 27155 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:09.502701 27155 catalog_manager.cc:1357] Generated new cluster ID: 30bf3de0e38b40a297ea59a320bed6ac
I20260320 02:58:09.502803 27155 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:09.533196 27155 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:09.535068 27155 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:09.548370 27155 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0975caafd9d44184a56b846d4cd6b31f: Generated new TSK 0
I20260320 02:58:09.550115 27155 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:09.642477 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:38405
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:09.781216 27169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:09.781471 27169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:09.781531 27169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:09.785094 27169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:09.785198 27169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:09.785311 27169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:09.790359 27169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:38405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:09.791841 27169 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:09.794533 27169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:09.803045 27177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:09.803108 27174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:09.803045 27175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:09.804273 27169 server_base.cc:1061] running on GCE node
I20260320 02:58:09.804680 27169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:09.805425 27169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:09.806618 27169 hybrid_clock.cc:648] HybridClock initialized: now 1773975489806563 us; error 59 us; skew 500 ppm
Mar 20 02:58:09 dist-test-slave-v3hj krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:09.812053 27169 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:58:09.813567 27169 webserver.cc:492] Webserver started at http://127.23.132.65:44857/ using document root <none> and password file <none>
I20260320 02:58:09.814340 27169 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:09.814478 27169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:09.815018 27169 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:09.818728 27169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "390e428c8f2b446182dc1ebe5073d9fb"
format_stamp: "Formatted at 2026-03-20 02:58:09 on dist-test-slave-v3hj"
server_key: "0fbd776e8a7954eb886aebc8bd4ba765"
server_key_iv: "0e97d8840c850ffde072b05f91bdbbfa"
server_key_version: "encryptionkey@0"
I20260320 02:58:09.819855 27169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "390e428c8f2b446182dc1ebe5073d9fb"
format_stamp: "Formatted at 2026-03-20 02:58:09 on dist-test-slave-v3hj"
server_key: "0fbd776e8a7954eb886aebc8bd4ba765"
server_key_iv: "0e97d8840c850ffde072b05f91bdbbfa"
server_key_version: "encryptionkey@0"
I20260320 02:58:09.824842 27169 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260320 02:58:09.828001 27184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:09.829560 27169 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:09.829753 27169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "390e428c8f2b446182dc1ebe5073d9fb"
format_stamp: "Formatted at 2026-03-20 02:58:09 on dist-test-slave-v3hj"
server_key: "0fbd776e8a7954eb886aebc8bd4ba765"
server_key_iv: "0e97d8840c850ffde072b05f91bdbbfa"
server_key_version: "encryptionkey@0"
I20260320 02:58:09.830503 27169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:09.860507 27169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:09.866511 27169 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:09.866797 27169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:09.867506 27169 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:09.868518 27169 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:09.868570 27169 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:09.868634 27169 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:09.868660 27169 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:09.882979 27169 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:37763
I20260320 02:58:09.883468 27297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:37763 every 8 connection(s)
I20260320 02:58:09.883972 27169 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:09.892345 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27169
I20260320 02:58:09.892480 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:09.892752 24081 external_mini_cluster.cc:1468] Setting key 25975d44a0537ec1a240c1e297618d4f
Mar 20 02:58:09 dist-test-slave-v3hj krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:09.906265 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:09.888232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:40883 (local address 127.23.132.126:38405)
0320 02:58:09.888392 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:09.888396 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:09.889425 (+ 1029us) server_negotiation.cc:408] Connection header received
0320 02:58:09.890717 (+ 1292us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:09.890722 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:09.890807 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:09.891083 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:09.894193 (+ 3110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:09.894924 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:09.895886 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:09.896121 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:09.900113 (+ 3992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:09.900138 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:09.900141 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:09.900173 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:09.902505 (+ 2332us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:09.903333 (+ 828us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:09.903339 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:09.903340 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:09.903617 (+ 277us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:09.904550 (+ 933us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:09.904559 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:09.904560 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:09.904879 (+ 319us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:09.905009 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:09.905983 (+ 974us) server_negotiation.cc:300] Negotiation successful
0320 02:58:09.906127 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:58:09.907939 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:09.888432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:38405 (local address 127.23.132.65:40883)
0320 02:58:09.889094 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:09.889161 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:09.890283 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:09.891337 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:09.891347 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:09.892078 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:09.893607 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:09.893691 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:09.895116 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:09.895120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:09.895661 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:09.895672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:09.895904 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:09.897658 (+ 1754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:09.897699 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:09.899881 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:09.902669 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:09.902678 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:09.902695 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:09.903087 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:09.903907 (+ 820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:09.903912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:09.903914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:09.904162 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:09.905120 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:09.905128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:09.905515 (+ 387us) client_negotiation.cc:769] Sending connection context
0320 02:58:09.905764 (+ 249us) client_negotiation.cc:240] Negotiation successful
0320 02:58:09.906913 (+ 1149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":132,"threads_started":1}
I20260320 02:58:09.910194 27298 heartbeater.cc:344] Connected to a master server at 127.23.132.126:38405
I20260320 02:58:09.910557 27298 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:09.911845 27298 heartbeater.cc:507] Master 127.23.132.126:38405 requested a full tablet report, sending...
I20260320 02:58:09.914713 27105 ts_manager.cc:194] Registered new tserver with Master: 390e428c8f2b446182dc1ebe5073d9fb (127.23.132.65:37763)
I20260320 02:58:09.915937 27105 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:40883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
W20260320 02:58:10.103143 24081 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.23.132.66: real 0.208s user 0.000s sys 0.004s
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:10.147768 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:38405
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:10.291594 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:10.291868 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:10.291942 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:10.296622 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:10.296789 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:10.296883 27305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:10.304618 27305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:38405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:10.306391 27305 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:10.307365 27305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:10.318130 27310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:10.318113 27313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:10.318141 27311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:10.318630 27305 server_base.cc:1061] running on GCE node
I20260320 02:58:10.321447 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:10.322733 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:10.324113 27305 hybrid_clock.cc:648] HybridClock initialized: now 1773975490324076 us; error 67 us; skew 500 ppm
Mar 20 02:58:10 dist-test-slave-v3hj krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:10.329140 27305 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:58:10.331231 27305 webserver.cc:492] Webserver started at http://127.23.132.66:34467/ using document root <none> and password file <none>
I20260320 02:58:10.332051 27305 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:10.332127 27305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:10.332336 27305 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:10.334223 27305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "0137c11c9fb44fd487dac10436a4d65e"
format_stamp: "Formatted at 2026-03-20 02:58:10 on dist-test-slave-v3hj"
server_key: "52734ebe86b7707a3b7e0e0a60f01016"
server_key_iv: "5e820cbf1efb17b205c6a7f239f78f40"
server_key_version: "encryptionkey@0"
I20260320 02:58:10.334707 27305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "0137c11c9fb44fd487dac10436a4d65e"
format_stamp: "Formatted at 2026-03-20 02:58:10 on dist-test-slave-v3hj"
server_key: "52734ebe86b7707a3b7e0e0a60f01016"
server_key_iv: "5e820cbf1efb17b205c6a7f239f78f40"
server_key_version: "encryptionkey@0"
I20260320 02:58:10.338601 27305 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:58:10.341781 27320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:10.343617 27305 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260320 02:58:10.343755 27305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "0137c11c9fb44fd487dac10436a4d65e"
format_stamp: "Formatted at 2026-03-20 02:58:10 on dist-test-slave-v3hj"
server_key: "52734ebe86b7707a3b7e0e0a60f01016"
server_key_iv: "5e820cbf1efb17b205c6a7f239f78f40"
server_key_version: "encryptionkey@0"
I20260320 02:58:10.343926 27305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:10.371116 27305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:10.379216 27305 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:10.379472 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:10.380297 27305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:10.382022 27305 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:10.382104 27305 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:10.382169 27305 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:10.382189 27305 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:10.402209 27305 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:35097
I20260320 02:58:10.402225 27435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:35097 every 8 connection(s)
I20260320 02:58:10.403537 27305 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:10.410441 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27305
I20260320 02:58:10.410588 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:10.410928 24081 external_mini_cluster.cc:1468] Setting key 78596494ac9d5a50115424204ada3a3c
Mar 20 02:58:10 dist-test-slave-v3hj krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:10.426191 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:10.405873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:56213 (local address 127.23.132.126:38405)
0320 02:58:10.406061 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:10.406199 (+ 138us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:10.406549 (+ 350us) server_negotiation.cc:408] Connection header received
0320 02:58:10.407513 (+ 964us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:10.407517 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:10.407569 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:10.407680 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:10.412029 (+ 4349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.412577 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:10.415172 (+ 2595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.415362 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:10.419142 (+ 3780us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:10.419179 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:10.419182 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:10.419217 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:10.421521 (+ 2304us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:10.422301 (+ 780us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:10.422306 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:10.422308 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:10.422541 (+ 233us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:10.423969 (+ 1428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:10.424253 (+ 284us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:10.424259 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:10.424463 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:10.424568 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:10.425758 (+ 1190us) server_negotiation.cc:300] Negotiation successful
0320 02:58:10.426057 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:58:10.426928 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:10.405779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:38405 (local address 127.23.132.66:56213)
0320 02:58:10.406420 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:10.406454 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:10.407299 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:10.410335 (+ 3036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:10.410349 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:10.411054 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:10.411763 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:10.411779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.413079 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:10.413082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:10.414901 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:10.414919 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.415301 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:10.416845 (+ 1544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:10.416888 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:10.418862 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:10.421747 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:10.421758 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:10.421770 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:10.422101 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:10.422688 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:10.422692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:10.422694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:10.422849 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:10.424771 (+ 1922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:10.424779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:10.425228 (+ 449us) client_negotiation.cc:769] Sending connection context
0320 02:58:10.425492 (+ 264us) client_negotiation.cc:240] Negotiation successful
0320 02:58:10.425866 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":252,"threads_started":1}
I20260320 02:58:10.428460 27436 heartbeater.cc:344] Connected to a master server at 127.23.132.126:38405
I20260320 02:58:10.428686 27436 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:10.429270 27436 heartbeater.cc:507] Master 127.23.132.126:38405 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
I20260320 02:58:10.431736 27105 ts_manager.cc:194] Registered new tserver with Master: 0137c11c9fb44fd487dac10436a4d65e (127.23.132.66:35097)
I20260320 02:58:10.432523 27105 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:56213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:10.495788 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:38405
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:10.653400 27443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:10.653692 27443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:10.654325 27443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:10.659860 27443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:10.659981 27443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:10.660099 27443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:10.667156 27443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:38405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:10.669024 27443 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:10.670498 27443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:10.680611 27448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:10.680621 27451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:10.681249 27443 server_base.cc:1061] running on GCE node
W20260320 02:58:10.680707 27449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:10.682562 27443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:10.683471 27443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:10.685097 27443 hybrid_clock.cc:648] HybridClock initialized: now 1773975490684909 us; error 40 us; skew 500 ppm
Mar 20 02:58:10 dist-test-slave-v3hj krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:10.690178 27443 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:58:10.693497 27443 webserver.cc:492] Webserver started at http://127.23.132.67:42171/ using document root <none> and password file <none>
I20260320 02:58:10.694639 27443 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:10.694736 27443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:10.694936 27443 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:10.696717 27443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "53632f3170bf4ae192bf08aa711cc519"
format_stamp: "Formatted at 2026-03-20 02:58:10 on dist-test-slave-v3hj"
server_key: "38a2e65f481fda980217a87180fae5fb"
server_key_iv: "d419ed38f62bdf0f7506c906b152db7d"
server_key_version: "encryptionkey@0"
I20260320 02:58:10.697212 27443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "53632f3170bf4ae192bf08aa711cc519"
format_stamp: "Formatted at 2026-03-20 02:58:10 on dist-test-slave-v3hj"
server_key: "38a2e65f481fda980217a87180fae5fb"
server_key_iv: "d419ed38f62bdf0f7506c906b152db7d"
server_key_version: "encryptionkey@0"
I20260320 02:58:10.701365 27443 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260320 02:58:10.704524 27458 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:10.707234 27443 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:10.707379 27443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "53632f3170bf4ae192bf08aa711cc519"
format_stamp: "Formatted at 2026-03-20 02:58:10 on dist-test-slave-v3hj"
server_key: "38a2e65f481fda980217a87180fae5fb"
server_key_iv: "d419ed38f62bdf0f7506c906b152db7d"
server_key_version: "encryptionkey@0"
I20260320 02:58:10.707499 27443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:10.734793 27443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:10.738587 27443 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:10.738809 27443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:10.739444 27443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:10.740355 27443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:10.740402 27443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:10.740442 27443 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:10.740456 27443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:10.751528 27443 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:34979
I20260320 02:58:10.751576 27574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:34979 every 8 connection(s)
I20260320 02:58:10.752696 27443 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:10.756594 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27443
I20260320 02:58:10.756793 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:10.757159 24081 external_mini_cluster.cc:1468] Setting key 1288cc756235f0b2283d825baad0cfd1
Mar 20 02:58:10 dist-test-slave-v3hj krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:10.774874 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:10.756012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:54909 (local address 127.23.132.126:38405)
0320 02:58:10.756221 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:10.756226 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:10.757353 (+ 1127us) server_negotiation.cc:408] Connection header received
0320 02:58:10.759351 (+ 1998us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:10.759355 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:10.759410 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:10.759491 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:10.761120 (+ 1629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.762074 (+ 954us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:10.762917 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.763096 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:10.767258 (+ 4162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:10.767292 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:10.767295 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:10.767336 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:10.770341 (+ 3005us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:10.771389 (+ 1048us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:10.771393 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:10.771395 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:10.771611 (+ 216us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:10.772593 (+ 982us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:10.772597 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:10.772600 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:10.773012 (+ 412us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:10.773238 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:10.774424 (+ 1186us) server_negotiation.cc:300] Negotiation successful
0320 02:58:10.774730 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260320 02:58:10.775842 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:10.756401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:38405 (local address 127.23.132.67:54909)
0320 02:58:10.757056 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:10.757226 (+ 170us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:10.759061 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:10.759702 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:10.759722 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:10.760300 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:10.760921 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:10.760935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.762291 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:10.762295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:10.762691 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:10.762698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:10.762932 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:10.764327 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:10.764426 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:10.766785 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:10.770637 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:10.770646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:10.770659 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:10.771258 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:10.771857 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:10.771861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:10.771865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:10.772198 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:10.773289 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:10.773299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:10.773934 (+ 635us) client_negotiation.cc:769] Sending connection context
0320 02:58:10.774401 (+ 467us) client_negotiation.cc:240] Negotiation successful
0320 02:58:10.774798 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":147,"threads_started":1}
I20260320 02:58:10.777868 27575 heartbeater.cc:344] Connected to a master server at 127.23.132.126:38405
I20260320 02:58:10.778339 27575 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:10.778976 27575 heartbeater.cc:507] Master 127.23.132.126:38405 requested a full tablet report, sending...
I20260320 02:58:10.780170 27105 ts_manager.cc:194] Registered new tserver with Master: 53632f3170bf4ae192bf08aa711cc519 (127.23.132.67:34979)
I20260320 02:58:10.780802 27105 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:54909
I20260320 02:58:10.789404 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:58:10.918880 27298 heartbeater.cc:499] Master 127.23.132.126:38405 was elected leader, sending a full tablet report...
W20260320 02:58:10.967630 27144 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:10.962661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.23.132.126:38405)
0320 02:58:10.962852 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:10.962856 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:10.964041 (+ 1185us) server_negotiation.cc:408] Connection header received
0320 02:58:10.965163 (+ 1122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:10.965167 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:10.965250 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:10.965368 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:10.967448 (+ 2080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48620: BlockingRecv error: recv got EOF from 127.0.0.1:48620 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260320 02:58:10.970569 27144 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:10.968632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.23.132.126:38405)
0320 02:58:10.968829 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:10.968834 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:10.968851 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:58:10.969060 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:10.969063 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:10.969286 (+ 223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:10.969482 (+ 196us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:10.970430 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48626: BlockingRecv error: recv got EOF from 127.0.0.1:48626 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260320 02:58:10.972481 27144 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:10.971178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.23.132.126:38405)
0320 02:58:10.971308 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:10.971313 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:10.971479 (+ 166us) server_negotiation.cc:408] Connection header received
0320 02:58:10.971781 (+ 302us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:10.971784 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:10.971833 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:10.971909 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:10.972347 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48636: BlockingRecv error: recv got EOF from 127.0.0.1:48636 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260320 02:58:10.978456 27144 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:10.976600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.23.132.126:38405)
0320 02:58:10.977047 (+ 447us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:10.977051 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:10.977064 (+ 13us) server_negotiation.cc:408] Connection header received
0320 02:58:10.977106 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:10.977108 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:10.977307 (+ 199us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:10.977425 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:10.978328 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48644: BlockingRecv error: recv got EOF from 127.0.0.1:48644 (error 108)
Metrics: {"server-negotiator.queue_time_us":356}
I20260320 02:58:10.991093 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27169
I20260320 02:58:10.999789 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27305
I20260320 02:58:11.007839 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27443
I20260320 02:58:11.019168 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27073
2026-03-20T02:58:11Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (4259 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:11 dist-test-slave-v3hj krb5kdc[27598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:11 dist-test-slave-v3hj krb5kdc[27598](info): set up 2 sockets
Mar 20 02:58:11 dist-test-slave-v3hj krb5kdc[27598](info): commencing operation
krb5kdc: starting...
W20260320 02:58:13.204212 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.158s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:13 dist-test-slave-v3hj krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:13.410646 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:33109
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:13.561617 27614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:13.563184 27614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:13.563264 27614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:13.568728 27614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:13.568816 27614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:13.568840 27614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:13.568859 27614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:13.568883 27614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:13.576743 27614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:13.578094 27614 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:13.579042 27614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:13.588268 27619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:13.588980 27622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:13.588392 27620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:13.589217 27614 server_base.cc:1061] running on GCE node
I20260320 02:58:13.590396 27614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:13.591920 27614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:13.595494 27614 hybrid_clock.cc:648] HybridClock initialized: now 1773975493595372 us; error 151 us; skew 500 ppm
Mar 20 02:58:13 dist-test-slave-v3hj krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975493, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:13.598775 27614 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:13.600747 27614 webserver.cc:492] Webserver started at http://127.23.132.126:35465/ using document root <none> and password file <none>
I20260320 02:58:13.602720 27614 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:13.602818 27614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:13.603200 27614 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:13.605849 27614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "ffd2c95023af4d36bc5302b4a13ed2f5"
format_stamp: "Formatted at 2026-03-20 02:58:13 on dist-test-slave-v3hj"
server_key: "575a2f15aec8dfa3419fcc6454e50ef3"
server_key_iv: "a7ea7254983c1d13ebbbec38056011d1"
server_key_version: "encryptionkey@0"
I20260320 02:58:13.606405 27614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "ffd2c95023af4d36bc5302b4a13ed2f5"
format_stamp: "Formatted at 2026-03-20 02:58:13 on dist-test-slave-v3hj"
server_key: "575a2f15aec8dfa3419fcc6454e50ef3"
server_key_iv: "a7ea7254983c1d13ebbbec38056011d1"
server_key_version: "encryptionkey@0"
I20260320 02:58:13.611976 27614 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260320 02:58:13.615522 27629 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:13.617621 27614 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:58:13.618284 27614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "ffd2c95023af4d36bc5302b4a13ed2f5"
format_stamp: "Formatted at 2026-03-20 02:58:13 on dist-test-slave-v3hj"
server_key: "575a2f15aec8dfa3419fcc6454e50ef3"
server_key_iv: "a7ea7254983c1d13ebbbec38056011d1"
server_key_version: "encryptionkey@0"
I20260320 02:58:13.618430 27614 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:13.641911 27614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:13.649063 27614 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:13.649250 27614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:13.664714 27614 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:33109
I20260320 02:58:13.664812 27681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:33109 every 8 connection(s)
I20260320 02:58:13.665951 27614 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:13.669615 27682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:13.670732 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27614
I20260320 02:58:13.670843 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:13.671540 24081 external_mini_cluster.cc:1468] Setting key 7d70053f84e2f5896bb5e64e7ecf24d9
I20260320 02:58:13.676960 27682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5: Bootstrap starting.
I20260320 02:58:13.679512 27682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:13.680262 27682 log.cc:826] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5: Log is configured to *not* fsync() on all Append() calls
Mar 20 02:58:13 dist-test-slave-v3hj krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:13.682752 27682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5: No bootstrap required, opened a new log
I20260320 02:58:13.686956 27682 raft_consensus.cc:359] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33109 } }
I20260320 02:58:13.688696 27682 raft_consensus.cc:385] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:13.688815 27682 raft_consensus.cc:740] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd2c95023af4d36bc5302b4a13ed2f5, State: Initialized, Role: FOLLOWER
I20260320 02:58:13.690353 27682 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33109 } }
I20260320 02:58:13.690622 27682 raft_consensus.cc:399] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:13.690713 27682 raft_consensus.cc:493] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:13.691267 27682 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:13.692922 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:13.672963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.23.132.126:33109)
0320 02:58:13.673424 (+ 461us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:13.673437 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:13.673475 (+ 38us) server_negotiation.cc:408] Connection header received
0320 02:58:13.674180 (+ 705us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:13.674205 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:13.674567 (+ 362us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:13.674940 (+ 373us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:13.676247 (+ 1307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:13.677387 (+ 1140us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:13.678316 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:13.678610 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:13.681239 (+ 2629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:13.681262 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:13.681281 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:13.681323 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:13.684617 (+ 3294us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:13.685331 (+ 714us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:13.685338 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:13.685345 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:13.685451 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:13.686045 (+ 594us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:13.686051 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:13.686053 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:13.687315 (+ 1262us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:13.688553 (+ 1238us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:13.689264 (+ 711us) server_negotiation.cc:300] Negotiation successful
0320 02:58:13.690255 (+ 991us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260320 02:58:13.694470 27682 raft_consensus.cc:515] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33109 } }
I20260320 02:58:13.694968 27682 leader_election.cc:304] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd2c95023af4d36bc5302b4a13ed2f5; no voters:
I20260320 02:58:13.695436 27682 leader_election.cc:290] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:13.695710 27687 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:13.696780 27687 raft_consensus.cc:697] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [term 1 LEADER]: Becoming Leader. State: Replica: ffd2c95023af4d36bc5302b4a13ed2f5, State: Running, Role: LEADER
I20260320 02:58:13.697204 27687 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33109 } }
I20260320 02:58:13.698540 27682 sys_catalog.cc:565] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:13.699760 27688 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33109 } } }
I20260320 02:58:13.699900 27688 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:13.700183 24081 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260320 02:58:13.700169 27689 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd2c95023af4d36bc5302b4a13ed2f5. Latest consensus state: current_term: 1 leader_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd2c95023af4d36bc5302b4a13ed2f5" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33109 } } }
I20260320 02:58:13.700254 27689 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:13.702979 27691 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:13.711607 27691 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:13.724296 27691 catalog_manager.cc:1357] Generated new cluster ID: 6dd94cf628dc4a6d9b8c9af8a8d13c4c
I20260320 02:58:13.724397 27691 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:13.731428 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:13.721757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.23.132.126:33109)
0320 02:58:13.721913 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:13.721918 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:13.722143 (+ 225us) server_negotiation.cc:408] Connection header received
0320 02:58:13.722809 (+ 666us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:13.722814 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:13.722875 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:13.723111 (+ 236us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:13.724092 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:13.726206 (+ 2114us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:13.728407 (+ 2201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:13.729401 (+ 994us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:13.729562 (+ 161us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:13.729650 (+ 88us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:13.729654 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:13.730324 (+ 670us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:13.730359 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:13.730376 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:13.730573 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:13.730856 (+ 283us) server_negotiation.cc:300] Negotiation successful
0320 02:58:13.731264 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260320 02:58:13.740994 27691 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:13.742707 27691 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:13.753477 27691 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ffd2c95023af4d36bc5302b4a13ed2f5: Generated new TSK 0
I20260320 02:58:13.754500 27691 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:58:13.766714 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27614
2026-03-20T02:58:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2750 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:13 dist-test-slave-v3hj krb5kdc[27715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:13 dist-test-slave-v3hj krb5kdc[27715](info): set up 2 sockets
Mar 20 02:58:13 dist-test-slave-v3hj krb5kdc[27715](info): commencing operation
krb5kdc: starting...
W20260320 02:58:16.004170 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.208s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:16 dist-test-slave-v3hj krb5kdc[27715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:16.198818 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:43555
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:43555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:16.341369 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:16.341645 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:16.341703 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:16.345705 27731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:16.345824 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:16.345849 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:16.345866 27731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:16.346662 27731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:16.353370 27731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:43555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:43555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:16.355381 27731 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:16.356328 27731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:16.363077 27736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:16.362959 27739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:16.363267 27737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:16.364377 27731 server_base.cc:1061] running on GCE node
I20260320 02:58:16.365432 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:16.366809 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:16.370239 27731 hybrid_clock.cc:648] HybridClock initialized: now 1773975496370191 us; error 64 us; skew 500 ppm
Mar 20 02:58:16 dist-test-slave-v3hj krb5kdc[27715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:16.373837 27731 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:16.376085 27731 webserver.cc:492] Webserver started at http://127.23.132.126:35369/ using document root <none> and password file <none>
I20260320 02:58:16.377074 27731 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:16.377135 27731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:16.377310 27731 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:16.379197 27731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "4c995aaa1b99489183cea88fc276086e"
format_stamp: "Formatted at 2026-03-20 02:58:16 on dist-test-slave-v3hj"
server_key: "c55fb1a50c967b3df2aaad1dcb10a44a"
server_key_iv: "c50fcfdad57e0b6e0b4e4f3fba30a07d"
server_key_version: "encryptionkey@0"
I20260320 02:58:16.379817 27731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "4c995aaa1b99489183cea88fc276086e"
format_stamp: "Formatted at 2026-03-20 02:58:16 on dist-test-slave-v3hj"
server_key: "c55fb1a50c967b3df2aaad1dcb10a44a"
server_key_iv: "c50fcfdad57e0b6e0b4e4f3fba30a07d"
server_key_version: "encryptionkey@0"
I20260320 02:58:16.384548 27731 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:58:16.387943 27746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:16.389638 27731 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:16.389905 27731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "4c995aaa1b99489183cea88fc276086e"
format_stamp: "Formatted at 2026-03-20 02:58:16 on dist-test-slave-v3hj"
server_key: "c55fb1a50c967b3df2aaad1dcb10a44a"
server_key_iv: "c50fcfdad57e0b6e0b4e4f3fba30a07d"
server_key_version: "encryptionkey@0"
I20260320 02:58:16.390275 27731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:16.412503 27731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:16.423686 27731 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:16.423864 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:16.437489 27798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:43555 every 8 connection(s)
I20260320 02:58:16.437595 27731 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:43555
I20260320 02:58:16.439400 27731 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:16.442767 27799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:16.447871 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27731
I20260320 02:58:16.448004 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:16.448237 24081 external_mini_cluster.cc:1468] Setting key ef759b8f26bc5117d8808737e13a8e60
I20260320 02:58:16.451248 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e: Bootstrap starting.
I20260320 02:58:16.453547 27799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:16.455401 27799 log.cc:826] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:16.458264 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e: No bootstrap required, opened a new log
Mar 20 02:58:16 dist-test-slave-v3hj krb5kdc[27715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:16.461226 27799 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c995aaa1b99489183cea88fc276086e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43555 } }
I20260320 02:58:16.461443 27799 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:16.461489 27799 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c995aaa1b99489183cea88fc276086e, State: Initialized, Role: FOLLOWER
I20260320 02:58:16.461977 27799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c995aaa1b99489183cea88fc276086e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43555 } }
I20260320 02:58:16.462144 27799 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:16.462203 27799 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:16.462410 27799 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:16.463586 27799 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c995aaa1b99489183cea88fc276086e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43555 } }
I20260320 02:58:16.464002 27799 leader_election.cc:304] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c995aaa1b99489183cea88fc276086e; no voters:
I20260320 02:58:16.464325 27799 leader_election.cc:290] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:16.464476 27804 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:16.464680 27804 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [term 1 LEADER]: Becoming Leader. State: Replica: 4c995aaa1b99489183cea88fc276086e, State: Running, Role: LEADER
I20260320 02:58:16.465036 27804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c995aaa1b99489183cea88fc276086e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43555 } }
I20260320 02:58:16.466493 27799 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:16.467231 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c995aaa1b99489183cea88fc276086e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c995aaa1b99489183cea88fc276086e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43555 } } }
I20260320 02:58:16.467428 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:16.467468 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:16.450256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.23.132.126:43555)
0320 02:58:16.450947 (+ 691us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:16.450968 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:16.451024 (+ 56us) server_negotiation.cc:408] Connection header received
0320 02:58:16.452012 (+ 988us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:16.452040 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:16.452528 (+ 488us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:16.453151 (+ 623us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:16.455029 (+ 1878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:16.455924 (+ 895us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:16.457343 (+ 1419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:16.457654 (+ 311us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:16.460621 (+ 2967us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:16.460651 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:16.460669 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:16.460725 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:16.463870 (+ 3145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:16.464792 (+ 922us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:16.464799 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:16.464805 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:16.464916 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:16.465229 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:16.465232 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:16.465234 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:16.465699 (+ 465us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:16.465940 (+ 241us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:16.466409 (+ 469us) server_negotiation.cc:300] Negotiation successful
0320 02:58:16.466709 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":146,"threads_started":1}
I20260320 02:58:16.468274 27808 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:16.470278 27805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c995aaa1b99489183cea88fc276086e. Latest consensus state: current_term: 1 leader_uuid: "4c995aaa1b99489183cea88fc276086e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c995aaa1b99489183cea88fc276086e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 43555 } } }
I20260320 02:58:16.470405 27805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:16.473664 27808 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:16.480656 27808 catalog_manager.cc:1357] Generated new cluster ID: 89ffb2a91b244681910e65cb2449b6eb
I20260320 02:58:16.480758 27808 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:16.494402 27808 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:16.496111 27808 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:16.504709 27808 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4c995aaa1b99489183cea88fc276086e: Generated new TSK 0
I20260320 02:58:16.505606 27808 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:16.610598 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:43555
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:16.781435 27827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:16.781680 27827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:16.781739 27827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:16.786448 27827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:16.786866 27827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:16.787513 27827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:16.793385 27827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:43555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:16.794564 27827 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:16.795805 27827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:16.805158 27832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:16.805667 27833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:16.805743 27835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:16.807358 27827 server_base.cc:1061] running on GCE node
I20260320 02:58:16.808312 27827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:16.809567 27827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:16.813156 27827 hybrid_clock.cc:648] HybridClock initialized: now 1773975496812997 us; error 171 us; skew 500 ppm
Mar 20 02:58:16 dist-test-slave-v3hj krb5kdc[27715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:16.822338 27827 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:58:16.823853 27827 webserver.cc:492] Webserver started at http://127.23.132.65:41095/ using document root <none> and password file <none>
I20260320 02:58:16.824455 27827 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:16.824514 27827 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:16.824714 27827 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:16.826740 27827 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "fbcf0184290c4bfb961e532014a57988"
format_stamp: "Formatted at 2026-03-20 02:58:16 on dist-test-slave-v3hj"
server_key: "a47b2a6bf25bf92dbc7dceff8cb6b30e"
server_key_iv: "3542d4098f16c954ff479f32822505d4"
server_key_version: "encryptionkey@0"
I20260320 02:58:16.827564 27827 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "fbcf0184290c4bfb961e532014a57988"
format_stamp: "Formatted at 2026-03-20 02:58:16 on dist-test-slave-v3hj"
server_key: "a47b2a6bf25bf92dbc7dceff8cb6b30e"
server_key_iv: "3542d4098f16c954ff479f32822505d4"
server_key_version: "encryptionkey@0"
I20260320 02:58:16.833877 27827 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260320 02:58:16.837563 27842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:16.840303 27827 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260320 02:58:16.841017 27827 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "fbcf0184290c4bfb961e532014a57988"
format_stamp: "Formatted at 2026-03-20 02:58:16 on dist-test-slave-v3hj"
server_key: "a47b2a6bf25bf92dbc7dceff8cb6b30e"
server_key_iv: "3542d4098f16c954ff479f32822505d4"
server_key_version: "encryptionkey@0"
I20260320 02:58:16.841176 27827 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:16.895382 27827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:16.901460 27827 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:16.901696 27827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:16.902514 27827 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:16.904006 27827 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:16.904072 27827 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:16.904134 27827 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:16.904158 27827 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:16.924629 27827 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:36235
I20260320 02:58:16.924665 27955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:36235 every 8 connection(s)
I20260320 02:58:16.926331 27827 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:16.934165 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27827
I20260320 02:58:16.934604 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:16.935379 24081 external_mini_cluster.cc:1468] Setting key 8e510041d871d3079657e4d5a69c9924
Mar 20 02:58:16 dist-test-slave-v3hj krb5kdc[27715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:16.947254 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:16.929187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:42897 (local address 127.23.132.126:43555)
0320 02:58:16.929601 (+ 414us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:16.929661 (+ 60us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:16.930831 (+ 1170us) server_negotiation.cc:408] Connection header received
0320 02:58:16.931843 (+ 1012us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:16.931847 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:16.931911 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:16.931992 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:16.933366 (+ 1374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:16.933931 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:16.934678 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:16.935191 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:16.939432 (+ 4241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:16.939456 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:16.939460 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:16.939496 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:16.943494 (+ 3998us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:16.944224 (+ 730us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:16.944229 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:16.944231 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:16.944297 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:16.944740 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:16.944743 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:16.944744 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:16.945652 (+ 908us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:16.945828 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:16.946467 (+ 639us) server_negotiation.cc:300] Negotiation successful
0320 02:58:16.946664 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260320 02:58:16.949065 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:16.929914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:43555 (local address 127.23.132.65:42897)
0320 02:58:16.930635 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:16.930678 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:16.931616 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:16.932181 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:16.932190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:16.932617 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:16.933200 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:16.933213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:16.934056 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:16.934060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:16.934435 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:16.934441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:16.934640 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:16.935845 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:16.935873 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:16.939219 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:16.943704 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:16.943713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:16.943727 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:16.944081 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:16.944501 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:16.944504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:16.944506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:16.944611 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:16.945837 (+ 1226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:16.945844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:16.946191 (+ 347us) client_negotiation.cc:769] Sending connection context
0320 02:58:16.946405 (+ 214us) client_negotiation.cc:240] Negotiation successful
0320 02:58:16.947138 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":297,"threads_started":1}
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:16.952361 27956 heartbeater.cc:344] Connected to a master server at 127.23.132.126:43555
I20260320 02:58:16.953655 27956 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:16.955462 27956 heartbeater.cc:507] Master 127.23.132.126:43555 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:58:16.957676 27763 ts_manager.cc:194] Registered new tserver with Master: fbcf0184290c4bfb961e532014a57988 (127.23.132.65:36235)
I20260320 02:58:16.959474 27763 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:42897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:17.022241 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:43555
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:17.194778 27963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:17.195132 27963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:17.195218 27963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:17.201283 27963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:17.201392 27963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:17.201506 27963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:17.208360 27963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:43555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:17.211403 27963 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:17.213199 27963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:17.221828 27969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:17.221854 27971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:17.221856 27968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:17.222193 27963 server_base.cc:1061] running on GCE node
I20260320 02:58:17.223156 27963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:17.223814 27963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:17.225039 27963 hybrid_clock.cc:648] HybridClock initialized: now 1773975497224992 us; error 64 us; skew 500 ppm
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[27715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:17.229164 27963 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:58:17.231128 27963 webserver.cc:492] Webserver started at http://127.23.132.66:37093/ using document root <none> and password file <none>
I20260320 02:58:17.231933 27963 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:17.232203 27963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:17.232692 27963 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:17.236230 27963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "5c0e0fbb3634496792ffa4048df74fb6"
format_stamp: "Formatted at 2026-03-20 02:58:17 on dist-test-slave-v3hj"
server_key: "5fb8651067d207262b8fc7fae70cdc6f"
server_key_iv: "29423cba1f196a379d499d58e31c09db"
server_key_version: "encryptionkey@0"
I20260320 02:58:17.236810 27963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "5c0e0fbb3634496792ffa4048df74fb6"
format_stamp: "Formatted at 2026-03-20 02:58:17 on dist-test-slave-v3hj"
server_key: "5fb8651067d207262b8fc7fae70cdc6f"
server_key_iv: "29423cba1f196a379d499d58e31c09db"
server_key_version: "encryptionkey@0"
I20260320 02:58:17.241591 27963 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260320 02:58:17.244659 27978 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:17.247375 27963 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:17.247622 27963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "5c0e0fbb3634496792ffa4048df74fb6"
format_stamp: "Formatted at 2026-03-20 02:58:17 on dist-test-slave-v3hj"
server_key: "5fb8651067d207262b8fc7fae70cdc6f"
server_key_iv: "29423cba1f196a379d499d58e31c09db"
server_key_version: "encryptionkey@0"
I20260320 02:58:17.247884 27963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:17.264483 27963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:17.269155 27963 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:17.269591 27963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:17.271030 27963 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:17.272382 27963 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:17.272580 27963 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:17.272735 27963 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:17.272763 27963 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:17.300140 27963 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:37781
I20260320 02:58:17.300163 28091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:37781 every 8 connection(s)
I20260320 02:58:17.301568 27963 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:17.302407 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 27963
I20260320 02:58:17.302587 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:17.303287 24081 external_mini_cluster.cc:1468] Setting key 75924f3a4df82d0c01a5edd0cd26f645
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[27715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:17.328173 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:17.305284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:42601 (local address 127.23.132.126:43555)
0320 02:58:17.305435 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:17.305439 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:17.306737 (+ 1298us) server_negotiation.cc:408] Connection header received
0320 02:58:17.309369 (+ 2632us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:17.309373 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:17.309464 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:17.310011 (+ 547us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:17.313095 (+ 3084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.314425 (+ 1330us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:17.316517 (+ 2092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.316706 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:17.321851 (+ 5145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:17.321883 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:17.321886 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:17.321919 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:17.324801 (+ 2882us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:17.325710 (+ 909us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:17.325713 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:17.325715 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:17.325778 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:17.326818 (+ 1040us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:17.326822 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:17.326824 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:17.327095 (+ 271us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:17.327189 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:17.327829 (+ 640us) server_negotiation.cc:300] Negotiation successful
0320 02:58:17.328003 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260320 02:58:17.328986 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:17.305621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:43555 (local address 127.23.132.66:42601)
0320 02:58:17.306431 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:17.306476 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:17.308454 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:17.310367 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:17.310383 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:17.311134 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:17.312597 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:17.312625 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.315895 (+ 3270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:17.315902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:17.316368 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:17.316380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.316665 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:17.318717 (+ 2052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:17.318750 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:17.321606 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:17.325200 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:17.325208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:17.325220 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:17.325547 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:17.326209 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:17.326214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:17.326216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:17.326499 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:17.327248 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:17.327255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:17.327539 (+ 284us) client_negotiation.cc:769] Sending connection context
0320 02:58:17.327772 (+ 233us) client_negotiation.cc:240] Negotiation successful
0320 02:58:17.328072 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":346,"threads_started":1}
I20260320 02:58:17.330974 28092 heartbeater.cc:344] Connected to a master server at 127.23.132.126:43555
I20260320 02:58:17.331234 28092 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:17.332158 28092 heartbeater.cc:507] Master 127.23.132.126:43555 requested a full tablet report, sending...
I20260320 02:58:17.333778 27763 ts_manager.cc:194] Registered new tserver with Master: 5c0e0fbb3634496792ffa4048df74fb6 (127.23.132.66:37781)
I20260320 02:58:17.334571 27763 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:42601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:17.392627 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:43555
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:38917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:17.546401 28099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:17.546689 28099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:17.547104 28099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:17.553401 28099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:17.553500 28099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:17.553606 28099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:17.558965 28099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:38917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:43555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:17.560225 28099 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:17.562469 28099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:17.572456 28104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:17.572592 28107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:17.572477 28105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:17.572497 28099 server_base.cc:1061] running on GCE node
I20260320 02:58:17.573441 28099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:17.574360 28099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:17.575589 28099 hybrid_clock.cc:648] HybridClock initialized: now 1773975497575541 us; error 77 us; skew 500 ppm
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[27715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:17.579384 28099 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:58:17.581184 28099 webserver.cc:492] Webserver started at http://127.23.132.67:40163/ using document root <none> and password file <none>
I20260320 02:58:17.581888 28099 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:17.581965 28099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:17.582146 28099 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:17.583925 28099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "3f651e5b69334135b78145778b7f1502"
format_stamp: "Formatted at 2026-03-20 02:58:17 on dist-test-slave-v3hj"
server_key: "c70cf62f631cc14d0292aff535bfeefd"
server_key_iv: "f06e8c55d796fe76b3d53981f094bc73"
server_key_version: "encryptionkey@0"
I20260320 02:58:17.584368 28099 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "3f651e5b69334135b78145778b7f1502"
format_stamp: "Formatted at 2026-03-20 02:58:17 on dist-test-slave-v3hj"
server_key: "c70cf62f631cc14d0292aff535bfeefd"
server_key_iv: "f06e8c55d796fe76b3d53981f094bc73"
server_key_version: "encryptionkey@0"
I20260320 02:58:17.589443 28099 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260320 02:58:17.592656 28114 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:17.596994 28099 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260320 02:58:17.597155 28099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "3f651e5b69334135b78145778b7f1502"
format_stamp: "Formatted at 2026-03-20 02:58:17 on dist-test-slave-v3hj"
server_key: "c70cf62f631cc14d0292aff535bfeefd"
server_key_iv: "f06e8c55d796fe76b3d53981f094bc73"
server_key_version: "encryptionkey@0"
I20260320 02:58:17.597291 28099 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:17.620479 28099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:17.624308 28099 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:17.624521 28099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:17.625584 28099 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:17.626941 28099 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:17.627015 28099 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:17.627070 28099 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:17.627099 28099 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:17.643848 28099 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:42385
I20260320 02:58:17.643989 28227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:42385 every 8 connection(s)
I20260320 02:58:17.645143 28099 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:17.654242 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28099
I20260320 02:58:17.654510 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:17.655231 24081 external_mini_cluster.cc:1468] Setting key ed26dc054936eb6728b885df1f95c4d7
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[27715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:17.669664 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:17.649869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:55791 (local address 127.23.132.126:43555)
0320 02:58:17.650670 (+ 801us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:17.650674 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:17.651810 (+ 1136us) server_negotiation.cc:408] Connection header received
0320 02:58:17.653675 (+ 1865us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:17.653681 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:17.653770 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:17.653926 (+ 156us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:17.657432 (+ 3506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.658344 (+ 912us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:17.659168 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.659365 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:17.663117 (+ 3752us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:17.663141 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:17.663144 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:17.663179 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:17.664649 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:17.665534 (+ 885us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:17.665537 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:17.665538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:17.665586 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:17.666522 (+ 936us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:17.666526 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:17.666527 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:17.666803 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:17.667592 (+ 789us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:17.669349 (+ 1757us) server_negotiation.cc:300] Negotiation successful
0320 02:58:17.669494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260320 02:58:17.670560 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:17.650539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:43555 (local address 127.23.132.67:55791)
0320 02:58:17.651506 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:17.651549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:17.652823 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:17.654341 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:17.654431 (+ 90us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:17.655493 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:17.657109 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:17.657135 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.658496 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:17.658500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:17.658949 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:17.658960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.659142 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:17.660579 (+ 1437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:17.660616 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:17.662714 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:17.664874 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:17.664883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:17.664908 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:17.665404 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:17.666181 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:17.666185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:17.666187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:17.666303 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:17.667620 (+ 1317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:17.667627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:17.668519 (+ 892us) client_negotiation.cc:769] Sending connection context
0320 02:58:17.668920 (+ 401us) client_negotiation.cc:240] Negotiation successful
0320 02:58:17.669314 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"spinlock_wait_cycles":335104,"thread_start_us":365,"threads_started":1}
I20260320 02:58:17.673496 28228 heartbeater.cc:344] Connected to a master server at 127.23.132.126:43555
I20260320 02:58:17.674576 28228 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:17.675609 28228 heartbeater.cc:507] Master 127.23.132.126:43555 requested a full tablet report, sending...
I20260320 02:58:17.677618 27763 ts_manager.cc:194] Registered new tserver with Master: 3f651e5b69334135b78145778b7f1502 (127.23.132.67:42385)
I20260320 02:58:17.678710 27763 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:55791
I20260320 02:58:17.686183 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[27715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975497, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[27715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975497, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:17.723066 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:17.712974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.23.132.126:43555)
0320 02:58:17.713286 (+ 312us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:17.713289 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:17.713416 (+ 127us) server_negotiation.cc:408] Connection header received
0320 02:58:17.713512 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:17.713515 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:17.713583 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:17.713668 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:17.714972 (+ 1304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.715498 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:17.716409 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:17.716556 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:17.718819 (+ 2263us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:17.718845 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:17.718848 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:17.718882 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:17.721164 (+ 2282us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:17.721664 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:17.721667 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:17.721669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:17.721760 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:17.722210 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:17.722213 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:17.722215 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:17.722422 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:17.722536 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:17.722801 (+ 265us) server_negotiation.cc:300] Negotiation successful
0320 02:58:17.722928 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260320 02:58:17.723508 27763 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51598
I20260320 02:58:17.724823 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27827
I20260320 02:58:17.736579 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27963
I20260320 02:58:17.743243 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28099
I20260320 02:58:17.750039 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 27731
2026-03-20T02:58:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3980 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[28240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[28240](info): set up 2 sockets
Mar 20 02:58:17 dist-test-slave-v3hj krb5kdc[28240](info): commencing operation
krb5kdc: starting...
W20260320 02:58:20.233106 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.456s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:20 dist-test-slave-v3hj krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:20.429580 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:42975
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:42975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:20.564972 28257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:20.566581 28257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:20.568086 28257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:20.573347 28257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:20.573640 28257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:20.574359 28257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:20.574402 28257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:20.574440 28257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:20.581161 28257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:42975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:42975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:20.583675 28257 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:20.584861 28257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:20.600343 28262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:20.600332 28265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:20.600818 28263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:20.601480 28257 server_base.cc:1061] running on GCE node
I20260320 02:58:20.602944 28257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:20.604075 28257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:20.605607 28257 hybrid_clock.cc:648] HybridClock initialized: now 1773975500605474 us; error 133 us; skew 500 ppm
Mar 20 02:58:20 dist-test-slave-v3hj krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:20.611654 28257 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:20.613009 28257 webserver.cc:492] Webserver started at http://127.23.132.126:36047/ using document root <none> and password file <none>
I20260320 02:58:20.613538 28257 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:20.613590 28257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:20.613765 28257 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:20.617126 28257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "2c3e73d67f844bd3ac80c14a74f167b9"
format_stamp: "Formatted at 2026-03-20 02:58:20 on dist-test-slave-v3hj"
server_key: "e1070f292000ab9653d0f854b2c1b689"
server_key_iv: "d8b7a352459028d5c6928b0688e47726"
server_key_version: "encryptionkey@0"
I20260320 02:58:20.617766 28257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "2c3e73d67f844bd3ac80c14a74f167b9"
format_stamp: "Formatted at 2026-03-20 02:58:20 on dist-test-slave-v3hj"
server_key: "e1070f292000ab9653d0f854b2c1b689"
server_key_iv: "d8b7a352459028d5c6928b0688e47726"
server_key_version: "encryptionkey@0"
I20260320 02:58:20.622196 28257 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:20.625272 28272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:20.626556 28257 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:20.626744 28257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "2c3e73d67f844bd3ac80c14a74f167b9"
format_stamp: "Formatted at 2026-03-20 02:58:20 on dist-test-slave-v3hj"
server_key: "e1070f292000ab9653d0f854b2c1b689"
server_key_iv: "d8b7a352459028d5c6928b0688e47726"
server_key_version: "encryptionkey@0"
I20260320 02:58:20.626870 28257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:20.645267 28257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:20.656450 28257 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:20.656630 28257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:20.664981 28257 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:42975
I20260320 02:58:20.664999 28324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:42975 every 8 connection(s)
I20260320 02:58:20.667317 28257 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:20.669246 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28257
I20260320 02:58:20.669389 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:20.669636 24081 external_mini_cluster.cc:1468] Setting key cb2d25030a2a81bc79fad27e98eb9ca3
I20260320 02:58:20.674306 28326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Mar 20 02:58:20 dist-test-slave-v3hj krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:20.681780 28326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9: Bootstrap starting.
I20260320 02:58:20.684813 28326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:20.686219 28326 log.cc:826] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:20.687255 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:20.672125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.23.132.126:42975)
0320 02:58:20.673017 (+ 892us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:20.673030 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:20.673072 (+ 42us) server_negotiation.cc:408] Connection header received
0320 02:58:20.673990 (+ 918us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:20.674017 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:20.674393 (+ 376us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:20.674769 (+ 376us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:20.676647 (+ 1878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:20.677987 (+ 1340us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:20.678711 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:20.679031 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:20.681084 (+ 2053us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:20.681158 (+ 74us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:20.681174 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:20.681236 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:20.683532 (+ 2296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:20.684029 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:20.684035 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:20.684041 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:20.684147 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:20.684432 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:20.684435 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:20.684437 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:20.684917 (+ 480us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:20.685177 (+ 260us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:20.685691 (+ 514us) server_negotiation.cc:300] Negotiation successful
0320 02:58:20.686223 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":285,"threads_started":1}
I20260320 02:58:20.690443 28326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9: No bootstrap required, opened a new log
I20260320 02:58:20.695389 28326 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42975 } }
I20260320 02:58:20.696193 28326 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:20.696410 28326 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e73d67f844bd3ac80c14a74f167b9, State: Initialized, Role: FOLLOWER
I20260320 02:58:20.697022 28326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42975 } }
I20260320 02:58:20.697173 28326 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:20.697242 28326 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:20.697358 28326 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:20.698370 28326 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42975 } }
I20260320 02:58:20.698715 28326 leader_election.cc:304] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c3e73d67f844bd3ac80c14a74f167b9; no voters:
I20260320 02:58:20.699090 28326 leader_election.cc:290] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:20.699692 28330 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:20.699882 28330 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3e73d67f844bd3ac80c14a74f167b9, State: Running, Role: LEADER
I20260320 02:58:20.700194 28330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42975 } }
I20260320 02:58:20.700699 28326 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:20.701650 28331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42975 } } }
I20260320 02:58:20.701912 28331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:20.701699 28332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c3e73d67f844bd3ac80c14a74f167b9. Latest consensus state: current_term: 1 leader_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c3e73d67f844bd3ac80c14a74f167b9" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42975 } } }
I20260320 02:58:20.702067 28332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:20.702780 28340 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:20.705709 28340 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:20.711997 28340 catalog_manager.cc:1357] Generated new cluster ID: 3e0c6cb343f64c339ff8d29a2cd1ecba
I20260320 02:58:20.712093 28340 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:20.737382 28340 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:20.738587 28340 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:20.748201 28340 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2c3e73d67f844bd3ac80c14a74f167b9: Generated new TSK 0
I20260320 02:58:20.749593 28340 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:20.835947 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:42975
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:20.962343 28354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:20.962596 28354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:20.962656 28354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:20.966497 28354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:20.966586 28354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:20.966708 28354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:20.971273 28354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:20.973284 28354 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:20.974224 28354 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:20.981072 28359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:20.981063 28362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:20.981063 28360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:20.981629 28354 server_base.cc:1061] running on GCE node
I20260320 02:58:20.982077 28354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:20.982697 28354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:20.983894 28354 hybrid_clock.cc:648] HybridClock initialized: now 1773975500983841 us; error 71 us; skew 500 ppm
Mar 20 02:58:20 dist-test-slave-v3hj krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:20.986521 28354 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:58:20.988077 28354 webserver.cc:492] Webserver started at http://127.23.132.65:44555/ using document root <none> and password file <none>
I20260320 02:58:20.989228 28354 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:20.989307 28354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:20.989531 28354 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:20.991761 28354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "b1a46f1370e74cc4b9c4f29cc9c4ed6e"
format_stamp: "Formatted at 2026-03-20 02:58:20 on dist-test-slave-v3hj"
server_key: "6acfdf8cfa9c80342da533f3a74ebd20"
server_key_iv: "8a182eefd48586b4101dc96e88dee473"
server_key_version: "encryptionkey@0"
I20260320 02:58:20.992883 28354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "b1a46f1370e74cc4b9c4f29cc9c4ed6e"
format_stamp: "Formatted at 2026-03-20 02:58:20 on dist-test-slave-v3hj"
server_key: "6acfdf8cfa9c80342da533f3a74ebd20"
server_key_iv: "8a182eefd48586b4101dc96e88dee473"
server_key_version: "encryptionkey@0"
I20260320 02:58:20.998914 28354 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260320 02:58:21.002012 28369 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.003446 28354 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260320 02:58:21.003572 28354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "b1a46f1370e74cc4b9c4f29cc9c4ed6e"
format_stamp: "Formatted at 2026-03-20 02:58:20 on dist-test-slave-v3hj"
server_key: "6acfdf8cfa9c80342da533f3a74ebd20"
server_key_iv: "8a182eefd48586b4101dc96e88dee473"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.003695 28354 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:21.019784 28354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:21.022903 28354 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:21.023097 28354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:21.023839 28354 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:21.024818 28354 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:21.024880 28354 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.024933 28354 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:21.024976 28354 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.046806 28354 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:45827
I20260320 02:58:21.047138 28484 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:45827 every 8 connection(s)
I20260320 02:58:21.048343 28354 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:21.055035 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28354
I20260320 02:58:21.055358 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:21.055706 24081 external_mini_cluster.cc:1468] Setting key 40e5f5a6d0b6aa1e078f19d98d64970a
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975500, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:21.066680 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.051597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:34627 (local address 127.23.132.126:42975)
0320 02:58:21.051994 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:21.051998 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:21.052937 (+ 939us) server_negotiation.cc:408] Connection header received
0320 02:58:21.053869 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:21.053872 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:21.053974 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:21.054054 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:21.056826 (+ 2772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.057345 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.058526 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.058710 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.061693 (+ 2983us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:21.061713 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:21.061716 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:21.061744 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:21.063694 (+ 1950us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.064514 (+ 820us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.064519 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.064521 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.064572 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.065102 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.065108 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.065110 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.065328 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:21.065442 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:21.066319 (+ 877us) server_negotiation.cc:300] Negotiation successful
0320 02:58:21.066486 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260320 02:58:21.067903 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.051536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42975 (local address 127.23.132.65:34627)
0320 02:58:21.052599 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:21.052649 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:21.053591 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:21.054476 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:21.054490 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:21.055163 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:21.056655 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:21.056674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.057589 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.057593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:21.058386 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:21.058398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.058590 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.059307 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:21.059334 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:21.061442 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:21.063962 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:21.063973 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:21.063994 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:21.064373 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:21.064667 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:21.064671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:21.064673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:21.064853 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:21.065474 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:21.065482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:21.066033 (+ 551us) client_negotiation.cc:769] Sending connection context
0320 02:58:21.066264 (+ 231us) client_negotiation.cc:240] Negotiation successful
0320 02:58:21.066601 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":155,"threads_started":1}
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:21.070693 28485 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42975
I20260320 02:58:21.070966 28485 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:21.072005 28485 heartbeater.cc:507] Master 127.23.132.126:42975 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:58:21.075269 28289 ts_manager.cc:194] Registered new tserver with Master: b1a46f1370e74cc4b9c4f29cc9c4ed6e (127.23.132.65:45827)
I20260320 02:58:21.077173 28289 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:34627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:21.135701 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:42975
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:21.282522 28492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:21.282768 28492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:21.282828 28492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:21.287847 28492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:21.287950 28492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:21.288069 28492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:21.297202 28492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:21.298491 28492 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:21.299399 28492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:21.308410 28500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:21.309077 28492 server_base.cc:1061] running on GCE node
W20260320 02:58:21.309163 28498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:21.309165 28497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:21.310269 28492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:21.311648 28492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:21.312916 28492 hybrid_clock.cc:648] HybridClock initialized: now 1773975501312891 us; error 45 us; skew 500 ppm
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:21.316838 28492 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:58:21.319191 28492 webserver.cc:492] Webserver started at http://127.23.132.66:42113/ using document root <none> and password file <none>
I20260320 02:58:21.319895 28492 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:21.320008 28492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:21.320201 28492 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:21.322144 28492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "3972277b2f1d4b5888130fa0bea7947c"
format_stamp: "Formatted at 2026-03-20 02:58:21 on dist-test-slave-v3hj"
server_key: "e06a174c2ff750db20fa750afba2a6db"
server_key_iv: "6a14d627b90c740fa30dee69483a7257"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.322620 28492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "3972277b2f1d4b5888130fa0bea7947c"
format_stamp: "Formatted at 2026-03-20 02:58:21 on dist-test-slave-v3hj"
server_key: "e06a174c2ff750db20fa750afba2a6db"
server_key_iv: "6a14d627b90c740fa30dee69483a7257"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.327265 28492 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:58:21.330662 28507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.334084 28492 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260320 02:58:21.334358 28492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "3972277b2f1d4b5888130fa0bea7947c"
format_stamp: "Formatted at 2026-03-20 02:58:21 on dist-test-slave-v3hj"
server_key: "e06a174c2ff750db20fa750afba2a6db"
server_key_iv: "6a14d627b90c740fa30dee69483a7257"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.334474 28492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:21.350955 28492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:21.356400 28492 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:21.356586 28492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:21.357424 28492 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:21.358417 28492 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:21.358469 28492 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.358521 28492 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:21.358551 28492 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.374943 28492 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:36073
I20260320 02:58:21.374959 28621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:36073 every 8 connection(s)
I20260320 02:58:21.376441 28492 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:21.385922 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28492
I20260320 02:58:21.386389 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:21.387017 24081 external_mini_cluster.cc:1468] Setting key ca403d6605dd7af10ad05f20d1888cf1
I20260320 02:58:21.396425 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.378809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:43123 (local address 127.23.132.126:42975)
0320 02:58:21.378985 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:21.378989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:21.379659 (+ 670us) server_negotiation.cc:408] Connection header received
0320 02:58:21.380534 (+ 875us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:21.380538 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:21.380602 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:21.380675 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:21.382431 (+ 1756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.383119 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.383788 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.384059 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.388805 (+ 4746us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:21.388825 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:21.388828 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:21.388854 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:21.392081 (+ 3227us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.393631 (+ 1550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.393635 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.393637 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.393705 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.394753 (+ 1048us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.394758 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.394759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.395111 (+ 352us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:21.395452 (+ 341us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:21.396074 (+ 622us) server_negotiation.cc:300] Negotiation successful
0320 02:58:21.396241 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260320 02:58:21.397051 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.379042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42975 (local address 127.23.132.66:43123)
0320 02:58:21.379504 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:21.379537 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:21.380336 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:21.380851 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:21.380860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:21.381272 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:21.382257 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:21.382273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.383264 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.383268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:21.383671 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:21.383678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.383961 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.384754 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:21.384781 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:21.388381 (+ 3600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:21.392435 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:21.392448 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:21.392468 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:21.393413 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:21.393838 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:21.393841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:21.393843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:21.394484 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:21.395407 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:21.395415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:21.395773 (+ 358us) client_negotiation.cc:769] Sending connection context
0320 02:58:21.395984 (+ 211us) client_negotiation.cc:240] Negotiation successful
0320 02:58:21.396289 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260320 02:58:21.398339 28622 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42975
I20260320 02:58:21.398571 28622 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:21.399530 28622 heartbeater.cc:507] Master 127.23.132.126:42975 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:58:21.401261 28289 ts_manager.cc:194] Registered new tserver with Master: 3972277b2f1d4b5888130fa0bea7947c (127.23.132.66:36073)
I20260320 02:58:21.401957 28289 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:43123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:21.479708 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:42975
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:21.644757 28629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:21.645020 28629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:21.645084 28629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:21.655458 28629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:21.655565 28629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:21.655702 28629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:21.661300 28629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:21.662483 28629 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:21.663326 28629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:21.675457 28635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:21.675469 28637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:21.675737 28634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:21.676034 28629 server_base.cc:1061] running on GCE node
I20260320 02:58:21.676636 28629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:21.677573 28629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:21.680948 28629 hybrid_clock.cc:648] HybridClock initialized: now 1773975501680897 us; error 67 us; skew 500 ppm
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:21.683846 28629 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:58:21.685009 28629 webserver.cc:492] Webserver started at http://127.23.132.67:45907/ using document root <none> and password file <none>
I20260320 02:58:21.687119 28629 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:21.687247 28629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:21.687943 28629 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:21.691221 28629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "df8865ddf84b4fd8bd21bdc0dc96858e"
format_stamp: "Formatted at 2026-03-20 02:58:21 on dist-test-slave-v3hj"
server_key: "0718aa0c08df1c728527544305b3274e"
server_key_iv: "9857881efafc197f7f62fa1f83470b10"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.692853 28629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "df8865ddf84b4fd8bd21bdc0dc96858e"
format_stamp: "Formatted at 2026-03-20 02:58:21 on dist-test-slave-v3hj"
server_key: "0718aa0c08df1c728527544305b3274e"
server_key_iv: "9857881efafc197f7f62fa1f83470b10"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.697405 28629 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260320 02:58:21.700172 28644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.701924 28629 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:21.702080 28629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "df8865ddf84b4fd8bd21bdc0dc96858e"
format_stamp: "Formatted at 2026-03-20 02:58:21 on dist-test-slave-v3hj"
server_key: "0718aa0c08df1c728527544305b3274e"
server_key_iv: "9857881efafc197f7f62fa1f83470b10"
server_key_version: "encryptionkey@0"
I20260320 02:58:21.702205 28629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:21.720461 28629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:21.728170 28629 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:21.728374 28629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:21.730083 28629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:21.732198 28629 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:21.732288 28629 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.732342 28629 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:21.732355 28629 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:21.757423 28629 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:32943
I20260320 02:58:21.757449 28758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:32943 every 8 connection(s)
I20260320 02:58:21.758497 28629 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:21.764565 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28629
I20260320 02:58:21.764773 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:21.765040 24081 external_mini_cluster.cc:1468] Setting key 2d32802622f53658af0d7e692f990d64
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:21.776849 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.760930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:40309 (local address 127.23.132.126:42975)
0320 02:58:21.761258 (+ 328us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:21.761262 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:21.762220 (+ 958us) server_negotiation.cc:408] Connection header received
0320 02:58:21.763250 (+ 1030us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:21.763316 (+ 66us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:21.763540 (+ 224us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:21.763766 (+ 226us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:21.766046 (+ 2280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.766621 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.767411 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.767588 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.771592 (+ 4004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:21.771689 (+ 97us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:21.771698 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:21.771788 (+ 90us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:21.774181 (+ 2393us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.775107 (+ 926us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.775111 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.775112 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.775166 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.775582 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.775585 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.775587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.775776 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:21.775869 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:21.776542 (+ 673us) server_negotiation.cc:300] Negotiation successful
0320 02:58:21.776663 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260320 02:58:21.777451 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.761175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42975 (local address 127.23.132.67:40309)
0320 02:58:21.762067 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:21.762106 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:21.763016 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:21.764019 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:21.764029 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:21.764525 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:21.765700 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:21.765716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.766741 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.766744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:21.767268 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:21.767279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.767515 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.768635 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:21.768672 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:21.771139 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:21.774354 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:21.774361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:21.774372 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:21.774655 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:21.775297 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:21.775300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:21.775302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:21.775433 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:21.775987 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:21.775993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:21.776253 (+ 260us) client_negotiation.cc:769] Sending connection context
0320 02:58:21.776433 (+ 180us) client_negotiation.cc:240] Negotiation successful
0320 02:58:21.776716 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":208,"threads_started":1}
I20260320 02:58:21.778719 28759 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42975
I20260320 02:58:21.778956 28759 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:21.779495 28759 heartbeater.cc:507] Master 127.23.132.126:42975 requested a full tablet report, sending...
I20260320 02:58:21.780706 28289 ts_manager.cc:194] Registered new tserver with Master: df8865ddf84b4fd8bd21bdc0dc96858e (127.23.132.67:32943)
I20260320 02:58:21.781459 28289 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:40309
I20260320 02:58:21.783151 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:58:21.819376 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.804457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.23.132.65:45827)
0320 02:58:21.805078 (+ 621us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:21.805084 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:21.805106 (+ 22us) server_negotiation.cc:408] Connection header received
0320 02:58:21.805159 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:21.805163 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:21.805337 (+ 174us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:21.805456 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:21.806867 (+ 1411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.808868 (+ 2001us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.810492 (+ 1624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.810821 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.814410 (+ 3589us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:21.814457 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:21.814474 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:21.814523 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:21.816993 (+ 2470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.817843 (+ 850us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.817849 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.817851 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.817916 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.818212 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.818215 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.818216 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.818426 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:21.818588 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:21.819008 (+ 420us) server_negotiation.cc:300] Negotiation successful
0320 02:58:21.819168 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":252,"threads_started":1}
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:21.836544 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:21.824018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.23.132.126:42975)
0320 02:58:21.824311 (+ 293us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:21.824315 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:21.824338 (+ 23us) server_negotiation.cc:408] Connection header received
0320 02:58:21.824397 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:21.824401 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:21.824474 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:21.824583 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:21.825417 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.826200 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:21.826858 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:21.827059 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:21.830109 (+ 3050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:21.830130 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:21.830133 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:21.830164 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:21.833309 (+ 3145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.834618 (+ 1309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.834623 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.834625 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.834690 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:21.835017 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:21.835020 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:21.835021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:21.835387 (+ 366us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:21.835539 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:21.836167 (+ 628us) server_negotiation.cc:300] Negotiation successful
0320 02:58:21.836379 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260320 02:58:21.837157 28288 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50792
I20260320 02:58:21.838445 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28354
I20260320 02:58:21.846839 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28492
I20260320 02:58:21.855161 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28629
I20260320 02:58:21.862694 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28257
2026-03-20T02:58:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4110 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28771](info): set up 2 sockets
Mar 20 02:58:21 dist-test-slave-v3hj krb5kdc[28771](info): commencing operation
krb5kdc: starting...
W20260320 02:58:24.033268 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.144s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:24 dist-test-slave-v3hj krb5kdc[28771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:24.226207 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:44587
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:44587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:24.364917 28787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:24.365198 28787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:24.365267 28787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:24.373348 28787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:24.373585 28787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:24.373658 28787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:24.373679 28787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:24.373696 28787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:24.379153 28787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:44587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:44587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:24.380292 28787 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:24.381136 28787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:24.387895 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:24.387899 28793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:24.388336 28792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:24.388801 28787 server_base.cc:1061] running on GCE node
I20260320 02:58:24.389451 28787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:24.390499 28787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:24.391789 28787 hybrid_clock.cc:648] HybridClock initialized: now 1773975504391669 us; error 142 us; skew 500 ppm
Mar 20 02:58:24 dist-test-slave-v3hj krb5kdc[28771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:24.396090 28787 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:24.396944 28787 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:24.397015 28787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:24.397207 28787 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:24.399021 28787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "720718d992924c25a059ad8e7b2d6f77"
format_stamp: "Formatted at 2026-03-20 02:58:24 on dist-test-slave-v3hj"
server_key: "3515cdfa421ccd87ed1efeb339beeb6e"
server_key_iv: "c0d90d45080e91a02ef05b8c5f699c57"
server_key_version: "encryptionkey@0"
I20260320 02:58:24.399488 28787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "720718d992924c25a059ad8e7b2d6f77"
format_stamp: "Formatted at 2026-03-20 02:58:24 on dist-test-slave-v3hj"
server_key: "3515cdfa421ccd87ed1efeb339beeb6e"
server_key_iv: "c0d90d45080e91a02ef05b8c5f699c57"
server_key_version: "encryptionkey@0"
I20260320 02:58:24.403116 28787 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260320 02:58:24.406484 28801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:24.408038 28787 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260320 02:58:24.408175 28787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "720718d992924c25a059ad8e7b2d6f77"
format_stamp: "Formatted at 2026-03-20 02:58:24 on dist-test-slave-v3hj"
server_key: "3515cdfa421ccd87ed1efeb339beeb6e"
server_key_iv: "c0d90d45080e91a02ef05b8c5f699c57"
server_key_version: "encryptionkey@0"
I20260320 02:58:24.408272 28787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:24.426399 28787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:24.430310 28787 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:24.430532 28787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:24.442777 28787 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:44587
I20260320 02:58:24.442791 28853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:44587 every 8 connection(s)
I20260320 02:58:24.443966 28787 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:24.444411 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28787
I20260320 02:58:24.444521 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:24.444818 24081 external_mini_cluster.cc:1468] Setting key 1f3fe7d06836e7adc734d4991394c144
I20260320 02:58:24.448359 28854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Mar 20 02:58:24 dist-test-slave-v3hj krb5kdc[28771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:24.456887 28854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: Bootstrap starting.
I20260320 02:58:24.462380 28854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:24.463642 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:24.446168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.23.132.126:44587)
0320 02:58:24.446533 (+ 365us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:24.446541 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:24.446573 (+ 32us) server_negotiation.cc:408] Connection header received
0320 02:58:24.448255 (+ 1682us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:24.448275 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:24.448578 (+ 303us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:24.448875 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:24.450013 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:24.450787 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:24.452236 (+ 1449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:24.452639 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:24.455994 (+ 3355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:24.456070 (+ 76us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:24.456083 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:24.456125 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:24.459965 (+ 3840us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:24.460602 (+ 637us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:24.460610 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:24.460616 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:24.460733 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:24.461175 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:24.461178 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:24.461180 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:24.461697 (+ 517us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:24.462018 (+ 321us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:24.462349 (+ 331us) server_negotiation.cc:300] Negotiation successful
0320 02:58:24.462775 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260320 02:58:24.464025 28854 log.cc:826] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:24.466969 28854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: No bootstrap required, opened a new log
I20260320 02:58:24.469448 28854 raft_consensus.cc:359] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720718d992924c25a059ad8e7b2d6f77" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44587 } }
I20260320 02:58:24.469650 28854 raft_consensus.cc:385] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:24.469710 28854 raft_consensus.cc:740] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720718d992924c25a059ad8e7b2d6f77, State: Initialized, Role: FOLLOWER
I20260320 02:58:24.470541 28854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720718d992924c25a059ad8e7b2d6f77" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44587 } }
I20260320 02:58:24.470772 28854 raft_consensus.cc:399] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:24.470839 28854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:24.470955 28854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:24.472297 28854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720718d992924c25a059ad8e7b2d6f77" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44587 } }
I20260320 02:58:24.472752 28854 leader_election.cc:304] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 720718d992924c25a059ad8e7b2d6f77; no voters:
I20260320 02:58:24.473464 28854 leader_election.cc:290] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:24.473937 28859 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:24.474282 28859 raft_consensus.cc:697] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [term 1 LEADER]: Becoming Leader. State: Replica: 720718d992924c25a059ad8e7b2d6f77, State: Running, Role: LEADER
I20260320 02:58:24.474759 28859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720718d992924c25a059ad8e7b2d6f77" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44587 } }
I20260320 02:58:24.475181 28854 sys_catalog.cc:565] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:24.476696 28861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 720718d992924c25a059ad8e7b2d6f77. Latest consensus state: current_term: 1 leader_uuid: "720718d992924c25a059ad8e7b2d6f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720718d992924c25a059ad8e7b2d6f77" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44587 } } }
I20260320 02:58:24.476718 28860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "720718d992924c25a059ad8e7b2d6f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720718d992924c25a059ad8e7b2d6f77" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44587 } } }
I20260320 02:58:24.476827 28861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:24.476837 28860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77 [sys.catalog]: This master's current role is: LEADER
W20260320 02:58:24.482398 28875 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:58:24.482512 28875 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:58:24.482801 28868 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:24.483868 28868 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:24.494300 28868 catalog_manager.cc:1357] Generated new cluster ID: bd113d8c46f6498d94e48a99638ddbab
I20260320 02:58:24.494445 28868 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:24.514456 28868 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:24.516119 28868 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:24.536841 28868 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: Generated new TSK 0
I20260320 02:58:24.538034 28868 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:24.639050 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:44587
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:24.776568 28882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:24.776849 28882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:24.776916 28882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:24.781435 28882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:24.781531 28882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:24.781620 28882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:24.787408 28882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:24.788523 28882 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:24.789358 28882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:24.798902 28888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:24.798902 28887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:24.799386 28890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:24.800347 28882 server_base.cc:1061] running on GCE node
I20260320 02:58:24.801766 28882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:24.803313 28882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:24.804564 28882 hybrid_clock.cc:648] HybridClock initialized: now 1773975504804521 us; error 48 us; skew 500 ppm
Mar 20 02:58:24 dist-test-slave-v3hj krb5kdc[28771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:24.808818 28882 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:58:24.810156 28882 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:24.810227 28882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:24.810422 28882 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:24.812248 28882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "7812bb3d91c54d7cb6a6f124bf8c9594"
format_stamp: "Formatted at 2026-03-20 02:58:24 on dist-test-slave-v3hj"
server_key: "fb3ce75fda196f8e5113760de013c43d"
server_key_iv: "bc1d76c5a2ad344de2fd5b5b730c1186"
server_key_version: "encryptionkey@0"
I20260320 02:58:24.813074 28882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "7812bb3d91c54d7cb6a6f124bf8c9594"
format_stamp: "Formatted at 2026-03-20 02:58:24 on dist-test-slave-v3hj"
server_key: "fb3ce75fda196f8e5113760de013c43d"
server_key_iv: "bc1d76c5a2ad344de2fd5b5b730c1186"
server_key_version: "encryptionkey@0"
I20260320 02:58:24.817914 28882 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260320 02:58:24.820484 28896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:24.822053 28882 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:24.822183 28882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "7812bb3d91c54d7cb6a6f124bf8c9594"
format_stamp: "Formatted at 2026-03-20 02:58:24 on dist-test-slave-v3hj"
server_key: "fb3ce75fda196f8e5113760de013c43d"
server_key_iv: "bc1d76c5a2ad344de2fd5b5b730c1186"
server_key_version: "encryptionkey@0"
I20260320 02:58:24.822386 28882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:24.841603 28882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:24.845080 28882 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:24.845276 28882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:24.846243 28882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:24.847270 28882 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:24.847322 28882 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:24.847375 28882 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:24.847404 28882 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:24.862139 28882 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:38215
I20260320 02:58:24.862156 29009 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:38215 every 8 connection(s)
I20260320 02:58:24.862891 28882 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:24.867177 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 28882
I20260320 02:58:24.867300 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:24.867717 24081 external_mini_cluster.cc:1468] Setting key d116cd75f03345a47b395c27ca39ee17
Mar 20 02:58:24 dist-test-slave-v3hj krb5kdc[28771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:24.882994 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:24.865461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:56533 (local address 127.23.132.126:44587)
0320 02:58:24.865717 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:24.865723 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:24.866915 (+ 1192us) server_negotiation.cc:408] Connection header received
0320 02:58:24.867762 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:24.867766 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:24.867833 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:24.868466 (+ 633us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:24.871811 (+ 3345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:24.872772 (+ 961us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:24.874781 (+ 2009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:24.875033 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:24.878456 (+ 3423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:24.878478 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:24.878481 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:24.878510 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:24.880473 (+ 1963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:24.881035 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:24.881038 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:24.881039 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:24.881086 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:24.881440 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:24.881444 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:24.881446 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:24.881680 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:24.881779 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:24.882681 (+ 902us) server_negotiation.cc:300] Negotiation successful
0320 02:58:24.882850 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260320 02:58:24.883447 29012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:24.865953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:44587 (local address 127.23.132.65:56533)
0320 02:58:24.866545 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:24.866590 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:24.867530 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:24.868800 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:24.868821 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:24.869606 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:24.871455 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:24.871474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:24.873130 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:24.873135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:24.874382 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:24.874396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:24.874933 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:24.875874 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:24.875906 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:24.878271 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:24.880630 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:24.880637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:24.880649 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:24.880917 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:24.881163 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:24.881167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:24.881168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:24.881274 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:24.881885 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:24.881890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:24.882236 (+ 346us) client_negotiation.cc:769] Sending connection context
0320 02:58:24.882431 (+ 195us) client_negotiation.cc:240] Negotiation successful
0320 02:58:24.882678 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":133,"threads_started":1}
I20260320 02:58:24.884879 29010 heartbeater.cc:344] Connected to a master server at 127.23.132.126:44587
I20260320 02:58:24.885174 29010 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:58:24.885818 29010 heartbeater.cc:507] Master 127.23.132.126:44587 requested a full tablet report, sending...
I20260320 02:58:24.887398 28818 ts_manager.cc:194] Registered new tserver with Master: 7812bb3d91c54d7cb6a6f124bf8c9594 (127.23.132.65:38215)
I20260320 02:58:24.888576 28818 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:56533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:24.942586 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:44587
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:25.082626 29017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:25.082950 29017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:25.083021 29017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:25.086827 29017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:25.087384 29017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:25.087709 29017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:25.092525 29017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:25.093670 29017 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:25.094549 29017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260320 02:58:25.101711 29017 server_base.cc:1061] running on GCE node
W20260320 02:58:25.101727 29023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:25.101739 29025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:25.101732 29022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:25.103174 29017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:25.104046 29017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:25.105324 29017 hybrid_clock.cc:648] HybridClock initialized: now 1773975505105310 us; error 28 us; skew 500 ppm
Mar 20 02:58:25 dist-test-slave-v3hj krb5kdc[28771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:25.108598 29017 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:58:25.109611 29017 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:25.109898 29017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:25.110227 29017 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:25.112428 29017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "188fd8ce08f94226903aefc67f66823e"
format_stamp: "Formatted at 2026-03-20 02:58:25 on dist-test-slave-v3hj"
server_key: "dee77a77786401c9cdca20e1054d6a4f"
server_key_iv: "30b50d23fa9dd5255449ca9330e9b605"
server_key_version: "encryptionkey@0"
I20260320 02:58:25.113030 29017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "188fd8ce08f94226903aefc67f66823e"
format_stamp: "Formatted at 2026-03-20 02:58:25 on dist-test-slave-v3hj"
server_key: "dee77a77786401c9cdca20e1054d6a4f"
server_key_iv: "30b50d23fa9dd5255449ca9330e9b605"
server_key_version: "encryptionkey@0"
I20260320 02:58:25.118673 29017 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260320 02:58:25.121275 29031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:25.122473 29017 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260320 02:58:25.122591 29017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "188fd8ce08f94226903aefc67f66823e"
format_stamp: "Formatted at 2026-03-20 02:58:25 on dist-test-slave-v3hj"
server_key: "dee77a77786401c9cdca20e1054d6a4f"
server_key_iv: "30b50d23fa9dd5255449ca9330e9b605"
server_key_version: "encryptionkey@0"
I20260320 02:58:25.122678 29017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:25.151521 29017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:25.156912 29017 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:25.157181 29017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:25.157776 29017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:25.159322 29017 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:25.159386 29017 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:25.159559 29017 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:25.159606 29017 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:25.178558 29017 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:36089
I20260320 02:58:25.178748 29144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:36089 every 8 connection(s)
I20260320 02:58:25.179468 29017 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:25.181035 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29017
I20260320 02:58:25.181133 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:25.181353 24081 external_mini_cluster.cc:1468] Setting key f4cd505d524e2be3e7e00acb2f674065
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
Mar 20 02:58:25 dist-test-slave-v3hj krb5kdc[28771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:25.203895 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.181999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:47729 (local address 127.23.132.126:44587)
0320 02:58:25.182149 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.182153 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.183470 (+ 1317us) server_negotiation.cc:408] Connection header received
0320 02:58:25.184336 (+ 866us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.184340 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.184394 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.184482 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:25.187492 (+ 3010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.188929 (+ 1437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.190901 (+ 1972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.191133 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.197851 (+ 6718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:25.197877 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:25.197879 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:25.197911 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:25.200120 (+ 2209us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:25.201242 (+ 1122us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:25.201245 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:25.201247 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:25.201303 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:25.202334 (+ 1031us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:25.202338 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:25.202340 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:25.202610 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:25.202703 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.203583 (+ 880us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.203762 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260320 02:58:25.204829 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.182816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:44587 (local address 127.23.132.66:47729)
0320 02:58:25.183339 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.183382 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.184162 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.184675 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.184687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.185260 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:25.186580 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.186600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.189500 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.189504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.190742 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.190754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.190996 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.191569 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:25.191596 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:25.197604 (+ 6008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:25.200281 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:25.200288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:25.200299 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:25.200966 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:25.201629 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:25.201635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:25.201637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:25.201888 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:25.202762 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:25.202770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:25.203184 (+ 414us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.203457 (+ 273us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.203940 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":125,"threads_started":1}
I20260320 02:58:25.208019 29145 heartbeater.cc:344] Connected to a master server at 127.23.132.126:44587
I20260320 02:58:25.208272 29145 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:25.208787 29145 heartbeater.cc:507] Master 127.23.132.126:44587 requested a full tablet report, sending...
I20260320 02:58:25.210417 28818 ts_manager.cc:194] Registered new tserver with Master: 188fd8ce08f94226903aefc67f66823e (127.23.132.66:36089)
I20260320 02:58:25.211081 28818 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:47729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:25.260295 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:44587
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:25.401049 29152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:25.401292 29152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:25.401355 29152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:25.407183 29152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:25.407274 29152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:25.407392 29152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:25.414927 29152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:25.416724 29152 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:25.418156 29152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:25.428459 29157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:25.428459 29158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:25.428612 29152 server_base.cc:1061] running on GCE node
W20260320 02:58:25.428539 29161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:25.430084 29152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:25.430776 29152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:25.434059 29152 hybrid_clock.cc:648] HybridClock initialized: now 1773975505433999 us; error 70 us; skew 500 ppm
Mar 20 02:58:25 dist-test-slave-v3hj krb5kdc[28771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:25.438017 29152 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:58:25.439647 29152 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:25.439827 29152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:25.440541 29152 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:25.442644 29152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "27ee87fea55a48679236acfb39bf52aa"
format_stamp: "Formatted at 2026-03-20 02:58:25 on dist-test-slave-v3hj"
server_key: "4dd7ea00c63715e08087119423362ce9"
server_key_iv: "f6bb9b47207fe1eaf72bf4b6ed82aba4"
server_key_version: "encryptionkey@0"
I20260320 02:58:25.443248 29152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "27ee87fea55a48679236acfb39bf52aa"
format_stamp: "Formatted at 2026-03-20 02:58:25 on dist-test-slave-v3hj"
server_key: "4dd7ea00c63715e08087119423362ce9"
server_key_iv: "f6bb9b47207fe1eaf72bf4b6ed82aba4"
server_key_version: "encryptionkey@0"
I20260320 02:58:25.449270 29152 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260320 02:58:25.452379 29167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:25.453740 29152 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:25.453920 29152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "27ee87fea55a48679236acfb39bf52aa"
format_stamp: "Formatted at 2026-03-20 02:58:25 on dist-test-slave-v3hj"
server_key: "4dd7ea00c63715e08087119423362ce9"
server_key_iv: "f6bb9b47207fe1eaf72bf4b6ed82aba4"
server_key_version: "encryptionkey@0"
I20260320 02:58:25.454039 29152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:25.496999 29152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:25.500929 29152 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:25.501094 29152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:25.501755 29152 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:25.502763 29152 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:25.502815 29152 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:25.502866 29152 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:25.502894 29152 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:25.520908 29152 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:33543
I20260320 02:58:25.520956 29282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:33543 every 8 connection(s)
I20260320 02:58:25.521938 29152 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:25.531286 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29152
I20260320 02:58:25.531448 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:25.531723 24081 external_mini_cluster.cc:1468] Setting key 67fdc02aec1d3fcaaaad3bbe091c06c3
Mar 20 02:58:25 dist-test-slave-v3hj krb5kdc[28771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:25.543016 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.524721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:44587 (local address 127.23.132.67:54293)
0320 02:58:25.525192 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.525238 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.526099 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.527343 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.527361 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.528121 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:25.530870 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.530897 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.532188 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.532192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.532917 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.532929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.533165 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.535036 (+ 1871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:25.535077 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:25.537271 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:25.539188 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:25.539194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:25.539204 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:25.539463 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:25.540079 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:25.540082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:25.540083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:25.540204 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:25.541265 (+ 1061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:25.541270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:25.541560 (+ 290us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.541867 (+ 307us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.542249 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":283,"threads_started":1}
I20260320 02:58:25.543030 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.524211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:54293 (local address 127.23.132.126:44587)
0320 02:58:25.524395 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.524399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.525379 (+ 980us) server_negotiation.cc:408] Connection header received
0320 02:58:25.526391 (+ 1012us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.526395 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.526480 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.527033 (+ 553us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:25.531173 (+ 4140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.531997 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.533054 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.533234 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.537553 (+ 4319us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:25.537572 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:25.537574 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:25.537603 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:25.539057 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:25.539875 (+ 818us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:25.539878 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:25.539879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:25.539940 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:25.540543 (+ 603us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:25.540548 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:25.540550 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:25.540865 (+ 315us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:25.541272 (+ 407us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.542634 (+ 1362us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.542858 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260320 02:58:25.546097 29283 heartbeater.cc:344] Connected to a master server at 127.23.132.126:44587
I20260320 02:58:25.546573 29283 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:25.547206 29283 heartbeater.cc:507] Master 127.23.132.126:44587 requested a full tablet report, sending...
I20260320 02:58:25.549698 28817 ts_manager.cc:194] Registered new tserver with Master: 27ee87fea55a48679236acfb39bf52aa (127.23.132.67:33543)
I20260320 02:58:25.550369 28817 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:54293
I20260320 02:58:25.552632 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:58:25.565953 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.555687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.23.132.126:44587)
0320 02:58:25.556034 (+ 347us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.556039 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.556146 (+ 107us) server_negotiation.cc:408] Connection header received
0320 02:58:25.556275 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.556279 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.556444 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.556545 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:25.557656 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.558223 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.558852 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.559046 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.560853 (+ 1807us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:25.560891 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:25.560894 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:25.560938 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:25.563104 (+ 2166us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:25.564175 (+ 1071us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:25.564180 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:25.564182 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:25.564236 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:25.564596 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:25.564660 (+ 64us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:25.564665 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:25.565113 (+ 448us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:25.565266 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.565488 (+ 222us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.565624 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260320 02:58:25.569183 28817 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:58:25.571599 28817 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:58:25.587584 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.582605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:33543 (local address 127.0.0.1:38658)
0320 02:58:25.583869 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.583882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.583981 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.584634 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.584638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.584656 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.584905 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.584912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.586007 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.586011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.586906 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.586916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.587025 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.587092 (+ 67us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.587182 (+ 90us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.587423 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"spinlock_wait_cycles":2304,"thread_start_us":191,"threads_started":1}
I20260320 02:58:25.588593 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.582553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38215 (local address 127.0.0.1:33328)
0320 02:58:25.583631 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.583668 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.584115 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.584648 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.584650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.584663 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.584905 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.584912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.586392 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.586395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.587304 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.587313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.588247 (+ 934us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.588264 (+ 17us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.588401 (+ 137us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.588464 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":74,"threads_started":1}
I20260320 02:58:25.589347 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.583224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.23.132.65:38215)
0320 02:58:25.583532 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.583538 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.583749 (+ 211us) server_negotiation.cc:408] Connection header received
0320 02:58:25.584256 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.584260 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.584381 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.584480 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.585028 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.586284 (+ 1256us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.587450 (+ 1166us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.588041 (+ 591us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.588168 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.588627 (+ 459us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.589191 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":62,"threads_started":1}
I20260320 02:58:25.589635 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.583151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.23.132.66:36089)
0320 02:58:25.583716 (+ 565us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.583722 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.583745 (+ 23us) server_negotiation.cc:408] Connection header received
0320 02:58:25.584178 (+ 433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.584184 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.584334 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.584457 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.584861 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.586205 (+ 1344us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.588036 (+ 1831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.589265 (+ 1229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.589352 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.589415 (+ 63us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.589517 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":119,"threads_started":1}
I20260320 02:58:25.589946 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.582468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:36089 (local address 127.0.0.1:49166)
0320 02:58:25.583630 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.583669 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.583987 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.584468 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.584474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.584496 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.584751 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.584758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.586345 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.586349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.587304 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.587313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.587422 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.587440 (+ 18us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.587486 (+ 46us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.587544 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":145,"threads_started":1}
I20260320 02:58:25.590103 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.582882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.23.132.67:33543)
0320 02:58:25.583917 (+ 1035us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.583922 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.583940 (+ 18us) server_negotiation.cc:408] Connection header received
0320 02:58:25.584225 (+ 285us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.584230 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.584381 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.584489 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.585063 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.585892 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.587409 (+ 1517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.589320 (+ 1911us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.589444 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.589516 (+ 72us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.589959 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":175,"threads_started":1}
I20260320 02:58:25.592543 29079 tablet_service.cc:1511] Processing CreateTablet for tablet d76d6f41302e43d0be59f304831186d2 (DEFAULT_TABLE table=test-table [id=79456bc4b39941eea3921390d71e9595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:25.592586 29215 tablet_service.cc:1511] Processing CreateTablet for tablet d76d6f41302e43d0be59f304831186d2 (DEFAULT_TABLE table=test-table [id=79456bc4b39941eea3921390d71e9595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:25.592586 28944 tablet_service.cc:1511] Processing CreateTablet for tablet d76d6f41302e43d0be59f304831186d2 (DEFAULT_TABLE table=test-table [id=79456bc4b39941eea3921390d71e9595]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:25.593641 28944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76d6f41302e43d0be59f304831186d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:25.593700 29215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76d6f41302e43d0be59f304831186d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:25.594117 29079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76d6f41302e43d0be59f304831186d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:25.602269 29302 tablet_bootstrap.cc:492] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Bootstrap starting.
I20260320 02:58:25.605718 29302 tablet_bootstrap.cc:654] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:25.606366 29303 tablet_bootstrap.cc:492] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: Bootstrap starting.
I20260320 02:58:25.609210 29303 tablet_bootstrap.cc:654] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:25.610167 29304 tablet_bootstrap.cc:492] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Bootstrap starting.
I20260320 02:58:25.610474 29302 log.cc:826] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:25.612185 29304 tablet_bootstrap.cc:654] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:25.613502 29304 log.cc:826] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:25.615126 29303 log.cc:826] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:25.616039 29302 tablet_bootstrap.cc:492] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: No bootstrap required, opened a new log
I20260320 02:58:25.616276 29302 ts_tablet_manager.cc:1403] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260320 02:58:25.617249 29304 tablet_bootstrap.cc:492] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: No bootstrap required, opened a new log
I20260320 02:58:25.617429 29304 ts_tablet_manager.cc:1403] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260320 02:58:25.619933 29303 tablet_bootstrap.cc:492] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: No bootstrap required, opened a new log
I20260320 02:58:25.620151 29303 ts_tablet_manager.cc:1403] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260320 02:58:25.620013 29302 raft_consensus.cc:359] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.620223 29302 raft_consensus.cc:385] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:25.620278 29302 raft_consensus.cc:740] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188fd8ce08f94226903aefc67f66823e, State: Initialized, Role: FOLLOWER
I20260320 02:58:25.620723 29302 consensus_queue.cc:260] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.621053 29304 raft_consensus.cc:359] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.621260 29304 raft_consensus.cc:385] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:25.621320 29304 raft_consensus.cc:740] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ee87fea55a48679236acfb39bf52aa, State: Initialized, Role: FOLLOWER
I20260320 02:58:25.621610 29145 heartbeater.cc:499] Master 127.23.132.126:44587 was elected leader, sending a full tablet report...
I20260320 02:58:25.621652 29302 ts_tablet_manager.cc:1434] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260320 02:58:25.621768 29304 consensus_queue.cc:260] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.622498 29283 heartbeater.cc:499] Master 127.23.132.126:44587 was elected leader, sending a full tablet report...
I20260320 02:58:25.622871 29304 ts_tablet_manager.cc:1434] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260320 02:58:25.623412 29303 raft_consensus.cc:359] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.623638 29303 raft_consensus.cc:385] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:25.623687 29303 raft_consensus.cc:740] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7812bb3d91c54d7cb6a6f124bf8c9594, State: Initialized, Role: FOLLOWER
I20260320 02:58:25.624236 29303 consensus_queue.cc:260] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.625335 29303 ts_tablet_manager.cc:1434] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260320 02:58:25.625686 29010 heartbeater.cc:499] Master 127.23.132.126:44587 was elected leader, sending a full tablet report...
W20260320 02:58:25.680929 29146 tablet.cc:2379] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:58:25.773003 29284 tablet.cc:2379] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:58:25.781526 29308 raft_consensus.cc:493] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:25.781754 29308 raft_consensus.cc:515] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.783363 29308 leader_election.cc:290] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27ee87fea55a48679236acfb39bf52aa (127.23.132.67:33543), 7812bb3d91c54d7cb6a6f124bf8c9594 (127.23.132.65:38215)
I20260320 02:58:25.787438 29309 raft_consensus.cc:493] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:25.787971 29309 raft_consensus.cc:515] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.790418 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.783643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:33543 (local address 127.23.132.66:46827)
0320 02:58:25.784417 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.784429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.784519 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.784849 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.784852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.784864 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.785169 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.785178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.787321 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.787334 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.788886 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.788902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.790017 (+ 1115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.790050 (+ 33us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.790163 (+ 113us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.790246 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":274,"thread_start_us":188,"threads_started":1}
I20260320 02:58:25.790500 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.783774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:46827 (local address 127.23.132.67:33543)
0320 02:58:25.783894 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.783899 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.784582 (+ 683us) server_negotiation.cc:408] Connection header received
0320 02:58:25.784645 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.784690 (+ 45us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.784750 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.784822 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.785408 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.786826 (+ 1418us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.789157 (+ 2331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.789855 (+ 698us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.789917 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.790259 (+ 342us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.790379 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260320 02:58:25.790521 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.783752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:59397 (local address 127.23.132.65:38215)
0320 02:58:25.783894 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.783899 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.784017 (+ 118us) server_negotiation.cc:408] Connection header received
0320 02:58:25.784411 (+ 394us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.784424 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.784632 (+ 208us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.784728 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.785408 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.786376 (+ 968us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.789160 (+ 2784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.789904 (+ 744us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.789962 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.790313 (+ 351us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.790421 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260320 02:58:25.790783 29309 leader_election.cc:290] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 188fd8ce08f94226903aefc67f66823e (127.23.132.66:36089), 7812bb3d91c54d7cb6a6f124bf8c9594 (127.23.132.65:38215)
I20260320 02:58:25.791019 29311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.783643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38215 (local address 127.23.132.66:59397)
0320 02:58:25.783918 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.783932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.784046 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.784783 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.784786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.784807 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.785169 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.785178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.787361 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.787364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.788994 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.789001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.790094 (+ 1093us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.790127 (+ 33us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.790738 (+ 611us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.790817 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260320 02:58:25.791970 29234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d76d6f41302e43d0be59f304831186d2" candidate_uuid: "188fd8ce08f94226903aefc67f66823e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ee87fea55a48679236acfb39bf52aa" is_pre_election: true
I20260320 02:58:25.792133 28964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d76d6f41302e43d0be59f304831186d2" candidate_uuid: "188fd8ce08f94226903aefc67f66823e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" is_pre_election: true
I20260320 02:58:25.792205 29234 raft_consensus.cc:2468] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188fd8ce08f94226903aefc67f66823e in term 0.
I20260320 02:58:25.792423 28964 raft_consensus.cc:2468] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188fd8ce08f94226903aefc67f66823e in term 0.
I20260320 02:58:25.792665 29033 leader_election.cc:304] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188fd8ce08f94226903aefc67f66823e, 27ee87fea55a48679236acfb39bf52aa; no voters:
I20260320 02:58:25.792933 29308 raft_consensus.cc:2804] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:58:25.792994 29308 raft_consensus.cc:493] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:25.793032 29308 raft_consensus.cc:3060] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:25.794421 29308 raft_consensus.cc:515] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.794791 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.790941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:36089 (local address 127.23.132.67:60577)
0320 02:58:25.791492 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.791507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.791601 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.792206 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.792209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.792226 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.792460 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.792468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.793371 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.793375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.794371 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.794383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.794534 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.794560 (+ 26us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.794612 (+ 52us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.794680 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459}
I20260320 02:58:25.794916 29308 leader_election.cc:290] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [CANDIDATE]: Term 1 election: Requested vote from peers 27ee87fea55a48679236acfb39bf52aa (127.23.132.67:33543), 7812bb3d91c54d7cb6a6f124bf8c9594 (127.23.132.65:38215)
I20260320 02:58:25.795382 28964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d76d6f41302e43d0be59f304831186d2" candidate_uuid: "188fd8ce08f94226903aefc67f66823e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594"
I20260320 02:58:25.795400 29234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d76d6f41302e43d0be59f304831186d2" candidate_uuid: "188fd8ce08f94226903aefc67f66823e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ee87fea55a48679236acfb39bf52aa"
I20260320 02:58:25.795501 28964 raft_consensus.cc:3060] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:25.795501 29234 raft_consensus.cc:3060] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:25.796360 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.791510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:60577 (local address 127.23.132.66:36089)
0320 02:58:25.791874 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.791878 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.791939 (+ 61us) server_negotiation.cc:408] Connection header received
0320 02:58:25.791985 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.791988 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.792039 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.792110 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.792584 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.793243 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.794677 (+ 1434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.796089 (+ 1412us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.796134 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.796190 (+ 56us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.796247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260320 02:58:25.796929 29234 raft_consensus.cc:2468] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188fd8ce08f94226903aefc67f66823e in term 1.
I20260320 02:58:25.796934 28964 raft_consensus.cc:2468] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188fd8ce08f94226903aefc67f66823e in term 1.
I20260320 02:58:25.796922 29099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d76d6f41302e43d0be59f304831186d2" candidate_uuid: "27ee87fea55a48679236acfb39bf52aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188fd8ce08f94226903aefc67f66823e" is_pre_election: true
I20260320 02:58:25.797137 29099 raft_consensus.cc:2393] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27ee87fea55a48679236acfb39bf52aa in current term 1: Already voted for candidate 188fd8ce08f94226903aefc67f66823e in this term.
I20260320 02:58:25.797251 29035 leader_election.cc:304] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188fd8ce08f94226903aefc67f66823e, 7812bb3d91c54d7cb6a6f124bf8c9594; no voters:
I20260320 02:58:25.797456 29308 raft_consensus.cc:2804] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:25.797966 29308 raft_consensus.cc:697] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 1 LEADER]: Becoming Leader. State: Replica: 188fd8ce08f94226903aefc67f66823e, State: Running, Role: LEADER
I20260320 02:58:25.798357 29308 consensus_queue.cc:237] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } }
I20260320 02:58:25.798733 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.791085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38215 (local address 127.23.132.67:55449)
0320 02:58:25.791570 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.791586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:25.791672 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:25.792282 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:25.792286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:25.792304 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:25.792593 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.792602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.794377 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.794382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:25.797941 (+ 3559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:25.797955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.798170 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.798195 (+ 25us) client_negotiation.cc:769] Sending connection context
0320 02:58:25.798246 (+ 51us) client_negotiation.cc:240] Negotiation successful
0320 02:58:25.798311 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":143,"threads_started":1}
I20260320 02:58:25.799129 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.791889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:55449 (local address 127.23.132.65:38215)
0320 02:58:25.792018 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.792022 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.792034 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:25.792071 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.792074 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.792117 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.792186 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:25.792720 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.793708 (+ 988us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.798194 (+ 4486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.798670 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.798714 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.798873 (+ 159us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.799022 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260320 02:58:25.800060 28964 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d76d6f41302e43d0be59f304831186d2" candidate_uuid: "27ee87fea55a48679236acfb39bf52aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" is_pre_election: true
I20260320 02:58:25.800244 28964 raft_consensus.cc:2393] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27ee87fea55a48679236acfb39bf52aa in current term 1: Already voted for candidate 188fd8ce08f94226903aefc67f66823e in this term.
I20260320 02:58:25.800700 29171 leader_election.cc:304] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27ee87fea55a48679236acfb39bf52aa; no voters: 188fd8ce08f94226903aefc67f66823e, 7812bb3d91c54d7cb6a6f124bf8c9594
I20260320 02:58:25.800946 29309 raft_consensus.cc:2749] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260320 02:58:25.802304 28817 catalog_manager.cc:5671] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e reported cstate change: term changed from 0 to 1, leader changed from <none> to 188fd8ce08f94226903aefc67f66823e (127.23.132.66). New cstate: current_term: 1 leader_uuid: "188fd8ce08f94226903aefc67f66823e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "188fd8ce08f94226903aefc67f66823e" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 } health_report { overall_health: UNKNOWN } } }
I20260320 02:58:25.834633 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.830681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.23.132.66:36089)
0320 02:58:25.830837 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:25.830842 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:25.830933 (+ 91us) server_negotiation.cc:408] Connection header received
0320 02:58:25.831016 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:25.831019 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:25.831078 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:25.831176 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:58:25.831536 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.831999 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:25.832803 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:25.833114 (+ 311us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:25.833389 (+ 275us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:58:25.834176 (+ 787us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:58:25.834328 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:25.834432 (+ 104us) server_negotiation.cc:300] Negotiation successful
0320 02:58:25.834498 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260320 02:58:25.841531 28964 raft_consensus.cc:1275] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 1 FOLLOWER]: Refusing update from remote peer 188fd8ce08f94226903aefc67f66823e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:25.841722 29234 raft_consensus.cc:1275] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 1 FOLLOWER]: Refusing update from remote peer 188fd8ce08f94226903aefc67f66823e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:25.842227 29314 consensus_queue.cc:1048] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7812bb3d91c54d7cb6a6f124bf8c9594" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 38215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:25.842428 29308 consensus_queue.cc:1048] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ee87fea55a48679236acfb39bf52aa" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:25.852530 29318 mvcc.cc:204] Tried to move back new op lower bound from 7266203671919296512 to 7266203671753908224. Current Snapshot: MvccSnapshot[applied={T|T < 7266203671919296512}]
I20260320 02:58:25.866559 28816 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:25.866837 28816 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:25.871330 28816 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 720718d992924c25a059ad8e7b2d6f77: Sending DeleteTablet for 3 replicas of tablet d76d6f41302e43d0be59f304831186d2
I20260320 02:58:25.872593 29215 tablet_service.cc:1558] Processing DeleteTablet for tablet d76d6f41302e43d0be59f304831186d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:25 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:38658
I20260320 02:58:25.872678 29079 tablet_service.cc:1558] Processing DeleteTablet for tablet d76d6f41302e43d0be59f304831186d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:25 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:49166
I20260320 02:58:25.873297 28944 tablet_service.cc:1558] Processing DeleteTablet for tablet d76d6f41302e43d0be59f304831186d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:25 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:33328
I20260320 02:58:25.873975 29330 tablet_replica.cc:333] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594: stopping tablet replica
I20260320 02:58:25.874732 29330 raft_consensus.cc:2243] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:25.875329 29330 raft_consensus.cc:2272] T d76d6f41302e43d0be59f304831186d2 P 7812bb3d91c54d7cb6a6f124bf8c9594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:25.876718 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28882
I20260320 02:58:25.882133 29328 tablet_replica.cc:333] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: stopping tablet replica
I20260320 02:58:25.882485 29328 raft_consensus.cc:2243] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:58:25.882520 29329 tablet_replica.cc:333] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: stopping tablet replica
I20260320 02:58:25.882906 29329 raft_consensus.cc:2243] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:25.883008 29328 raft_consensus.cc:2272] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:25.883487 29329 raft_consensus.cc:2272] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:25.884855 29329 ts_tablet_manager.cc:1916] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:25.884855 29328 ts_tablet_manager.cc:1916] T d76d6f41302e43d0be59f304831186d2 P 188fd8ce08f94226903aefc67f66823e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260320 02:58:25.887849 28805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.132.65:38215 (error 108)
I20260320 02:58:25.888828 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29017
I20260320 02:58:25.889621 29329 ts_tablet_manager.cc:1929] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:25.890184 29329 log.cc:1199] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/d76d6f41302e43d0be59f304831186d2
I20260320 02:58:25.890627 29329 ts_tablet_manager.cc:1950] T d76d6f41302e43d0be59f304831186d2 P 27ee87fea55a48679236acfb39bf52aa: Deleting consensus metadata
I20260320 02:58:25.892603 28803 catalog_manager.cc:5002] TS 27ee87fea55a48679236acfb39bf52aa (127.23.132.67:33543): tablet d76d6f41302e43d0be59f304831186d2 (table test-table [id=79456bc4b39941eea3921390d71e9595]) successfully deleted
I20260320 02:58:25.894088 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.893383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:38215 (local address 127.0.0.1:33332)
0320 02:58:25.893660 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.893765 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.65:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260320 02:58:25.894287 28805 catalog_manager.cc:4729] TS 7812bb3d91c54d7cb6a6f124bf8c9594 (127.23.132.65:38215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d76d6f41302e43d0be59f304831186d2: Network error: Client connection negotiation failed: client connection to 127.23.132.65:38215: connect: Connection refused (error 111)
I20260320 02:58:25.899497 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29152
I20260320 02:58:25.900651 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:25.900312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:36089 (local address 127.0.0.1:49178)
0320 02:58:25.900511 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:25.900561 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.66:36089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260320 02:58:25.905900 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 28787
2026-03-20T02:58:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4044 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-03-20T02:58:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:25Z Disabled control of system clock
I20260320 02:58:25.942179 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:36395
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260320 02:58:26.066821 29335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:26.067268 29335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:26.067341 29335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:26.071799 29335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:26.071918 29335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:26.071936 29335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:26.071955 29335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:26.071972 29335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:26.076825 29335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:36395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:26.078367 29335 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:26.079560 29335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:26.086139 29343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.086117 29340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.086122 29341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:26.086230 29335 server_base.cc:1061] running on GCE node
I20260320 02:58:26.087221 29335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:26.088425 29335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:26.089820 29335 hybrid_clock.cc:648] HybridClock initialized: now 1773975506089672 us; error 131 us; skew 500 ppm
I20260320 02:58:26.092993 29335 webserver.cc:492] Webserver started at http://127.23.132.126:42311/ using document root <none> and password file <none>
I20260320 02:58:26.094172 29335 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:26.094466 29335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:26.094969 29335 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:26.097020 29335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "9bc7dc6f16a345208ae4dd9cdadacf43"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "9154de684f7d180aea4de19e038bf7f6"
server_key_iv: "339810a2f1053a53867e19498ce51c08"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.097520 29335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "9bc7dc6f16a345208ae4dd9cdadacf43"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "9154de684f7d180aea4de19e038bf7f6"
server_key_iv: "339810a2f1053a53867e19498ce51c08"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.100998 29335 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260320 02:58:26.104202 29349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.105553 29335 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:26.105696 29335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "9bc7dc6f16a345208ae4dd9cdadacf43"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "9154de684f7d180aea4de19e038bf7f6"
server_key_iv: "339810a2f1053a53867e19498ce51c08"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.105952 29335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:26.112059 29335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:26.113183 29335 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:26.113400 29335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:26.121868 29335 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:36395
I20260320 02:58:26.121984 29401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:36395 every 8 connection(s)
I20260320 02:58:26.123489 29335 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:26.126578 29402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:26.130110 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29335
I20260320 02:58:26.130244 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:26.130520 24081 external_mini_cluster.cc:1468] Setting key bb7ef44265573220c067cbb429a1dddc
I20260320 02:58:26.132524 29402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: Bootstrap starting.
I20260320 02:58:26.134873 29402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:26.135707 29402 log.cc:826] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:26.137240 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.132129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.23.132.126:36395)
0320 02:58:26.132916 (+ 787us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.132927 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.132969 (+ 42us) server_negotiation.cc:408] Connection header received
0320 02:58:26.133636 (+ 667us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.133654 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.134087 (+ 433us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.134427 (+ 340us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.135440 (+ 1013us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.135446 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.135457 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.135469 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.135518 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.135529 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.135645 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.135946 (+ 301us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.136184 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":161,"threads_started":1}
I20260320 02:58:26.137929 29402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: No bootstrap required, opened a new log
I20260320 02:58:26.140640 29402 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 36395 } }
I20260320 02:58:26.140836 29402 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:26.140898 29402 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc7dc6f16a345208ae4dd9cdadacf43, State: Initialized, Role: FOLLOWER
I20260320 02:58:26.141260 29402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 36395 } }
I20260320 02:58:26.141376 29402 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:26.141434 29402 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:26.141534 29402 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:26.142542 29402 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 36395 } }
I20260320 02:58:26.142927 29402 leader_election.cc:304] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bc7dc6f16a345208ae4dd9cdadacf43; no voters:
I20260320 02:58:26.143241 29402 leader_election.cc:290] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:26.143428 29407 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:26.143761 29407 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc7dc6f16a345208ae4dd9cdadacf43, State: Running, Role: LEADER
I20260320 02:58:26.144120 29407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 36395 } }
I20260320 02:58:26.144349 29402 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:26.147138 29408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 36395 } } }
I20260320 02:58:26.147224 29409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bc7dc6f16a345208ae4dd9cdadacf43. Latest consensus state: current_term: 1 leader_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc7dc6f16a345208ae4dd9cdadacf43" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 36395 } } }
I20260320 02:58:26.147374 29408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:26.147374 29409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43 [sys.catalog]: This master's current role is: LEADER
W20260320 02:58:26.153039 29422 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:58:26.153234 29422 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:58:26.153373 29421 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:26.154595 29421 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:26.161183 29421 catalog_manager.cc:1357] Generated new cluster ID: 5cfbec1d7a80474696f17de3364cc73b
I20260320 02:58:26.161291 29421 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:26.181834 29421 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:26.182200 29421 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:26.192853 29421 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: Generated new TSK 0
I20260320 02:58:26.194198 29421 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:58:26.203518 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:36395
--builtin_ntp_servers=127.23.132.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260320 02:58:26.358199 29426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:26.358438 29426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:26.358500 29426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:26.362840 29426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:26.362926 29426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:26.363008 29426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:26.373499 29426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:36395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:26.374701 29426 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:26.375941 29426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:26.384995 29432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.385031 29431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.384995 29434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:26.385360 29426 server_base.cc:1061] running on GCE node
I20260320 02:58:26.387226 29426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:26.389272 29426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:26.392620 29426 hybrid_clock.cc:648] HybridClock initialized: now 1773975506392523 us; error 134 us; skew 500 ppm
I20260320 02:58:26.396534 29426 webserver.cc:492] Webserver started at http://127.23.132.65:41229/ using document root <none> and password file <none>
I20260320 02:58:26.397414 29426 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:26.397482 29426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:26.397754 29426 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:26.399799 29426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "fb756426f48140b298ef55403571f2fc"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "680a2bd0c01d19aa20d1da360ef6708f"
server_key_iv: "6408425a5adb3caaae746594d3b26556"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.400277 29426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "fb756426f48140b298ef55403571f2fc"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "680a2bd0c01d19aa20d1da360ef6708f"
server_key_iv: "6408425a5adb3caaae746594d3b26556"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.404989 29426 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260320 02:58:26.408834 29440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.410825 29426 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260320 02:58:26.411012 29426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "fb756426f48140b298ef55403571f2fc"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "680a2bd0c01d19aa20d1da360ef6708f"
server_key_iv: "6408425a5adb3caaae746594d3b26556"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.411248 29426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:26.417484 29426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:26.418314 29426 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:26.418509 29426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:26.419167 29426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:26.420464 29426 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:26.420533 29426 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.420594 29426 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:26.420624 29426 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.431238 29426 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:36313
I20260320 02:58:26.431562 29553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:36313 every 8 connection(s)
I20260320 02:58:26.432269 29426 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:26.433537 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29426
I20260320 02:58:26.433674 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:26.434020 24081 external_mini_cluster.cc:1468] Setting key 422001faea3733800afbf01c24dc5aa5
I20260320 02:58:26.436877 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:36395
--builtin_ntp_servers=127.23.132.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260320 02:58:26.441624 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.435695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:58927 (local address 127.23.132.126:36395)
0320 02:58:26.435872 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.435877 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.437665 (+ 1788us) server_negotiation.cc:408] Connection header received
0320 02:58:26.439018 (+ 1353us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.439021 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.439065 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.439147 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.440661 (+ 1514us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.440666 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.440669 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.440685 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.440713 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.440732 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.440842 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.441411 (+ 569us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.441468 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260320 02:58:26.442516 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.436326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:36395 (local address 127.23.132.65:58927)
0320 02:58:26.437492 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.437535 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.438594 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.439326 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.439334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.439423 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.440086 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.440087 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.440388 (+ 301us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.440458 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.441047 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.441057 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.441063 (+ 6us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.441424 (+ 361us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.441610 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":366,"threads_started":1}
I20260320 02:58:26.444231 29554 heartbeater.cc:344] Connected to a master server at 127.23.132.126:36395
I20260320 02:58:26.444679 29554 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:26.445211 29554 heartbeater.cc:507] Master 127.23.132.126:36395 requested a full tablet report, sending...
I20260320 02:58:26.448745 29363 ts_manager.cc:194] Registered new tserver with Master: fb756426f48140b298ef55403571f2fc (127.23.132.65:36313)
W20260320 02:58:26.595758 29557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:26.596110 29557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:26.596168 29557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:26.601445 29557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:26.601532 29557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:26.601900 29557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:26.608022 29557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:36395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:26.609910 29557 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:26.612552 29557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:26.623647 29563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.623631 29565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:26.623978 29557 server_base.cc:1061] running on GCE node
W20260320 02:58:26.623631 29562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:26.624917 29557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:26.625555 29557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:26.626806 29557 hybrid_clock.cc:648] HybridClock initialized: now 1773975506626779 us; error 40 us; skew 500 ppm
I20260320 02:58:26.630195 29557 webserver.cc:492] Webserver started at http://127.23.132.66:42363/ using document root <none> and password file <none>
I20260320 02:58:26.630861 29557 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:26.630936 29557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:26.631132 29557 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:26.636366 29557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "b1b8bdde5eee4d59978d0d3b81d39739"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "1919fea6b86b31114dc829e0b63d1ce9"
server_key_iv: "8a39cb155a34f12ecf1c2e68ac651552"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.636924 29557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "b1b8bdde5eee4d59978d0d3b81d39739"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "1919fea6b86b31114dc829e0b63d1ce9"
server_key_iv: "8a39cb155a34f12ecf1c2e68ac651552"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.640699 29557 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260320 02:58:26.643456 29571 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.644562 29557 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260320 02:58:26.644683 29557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "b1b8bdde5eee4d59978d0d3b81d39739"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "1919fea6b86b31114dc829e0b63d1ce9"
server_key_iv: "8a39cb155a34f12ecf1c2e68ac651552"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.644805 29557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:26.655985 29557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:26.656708 29557 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:26.657001 29557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:26.657627 29557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:26.658675 29557 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:26.658725 29557 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.658779 29557 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:26.658805 29557 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.678369 29557 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:41819
I20260320 02:58:26.678436 29684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:41819 every 8 connection(s)
I20260320 02:58:26.680559 29557 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:26.680861 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29557
I20260320 02:58:26.680998 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:26.681289 24081 external_mini_cluster.cc:1468] Setting key 3333d48c92411b3b67e203ca9c1736c3
I20260320 02:58:26.684823 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:36395
--builtin_ntp_servers=127.23.132.84:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260320 02:58:26.689144 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.683816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:44799 (local address 127.23.132.126:36395)
0320 02:58:26.683979 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.683983 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.684431 (+ 448us) server_negotiation.cc:408] Connection header received
0320 02:58:26.686486 (+ 2055us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.686491 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.686571 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.686671 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.688489 (+ 1818us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.688494 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.688496 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.688518 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.688544 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.688552 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.688671 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.688945 (+ 274us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.688999 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260320 02:58:26.690518 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.683468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:36395 (local address 127.23.132.66:44799)
0320 02:58:26.684305 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.684347 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.686010 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.686919 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.686932 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.687049 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.688104 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.688106 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.688145 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.688224 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.688737 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.688744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.688749 (+ 5us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.688971 (+ 222us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.689222 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":122,"threads_started":1}
I20260320 02:58:26.693547 29685 heartbeater.cc:344] Connected to a master server at 127.23.132.126:36395
I20260320 02:58:26.694460 29685 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:26.695019 29685 heartbeater.cc:507] Master 127.23.132.126:36395 requested a full tablet report, sending...
I20260320 02:58:26.696352 29363 ts_manager.cc:194] Registered new tserver with Master: b1b8bdde5eee4d59978d0d3b81d39739 (127.23.132.66:41819)
W20260320 02:58:26.823194 29688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:26.823519 29688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:26.823596 29688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:26.827292 29688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:26.827373 29688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:26.827451 29688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:26.832343 29688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:36395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:26.834698 29688 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:26.836531 29688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:26.844347 29696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.844290 29693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:26.844290 29694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:26.845028 29688 server_base.cc:1061] running on GCE node
I20260320 02:58:26.846033 29688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:26.846779 29688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:26.848088 29688 hybrid_clock.cc:648] HybridClock initialized: now 1773975506848050 us; error 98 us; skew 500 ppm
I20260320 02:58:26.850924 29688 webserver.cc:492] Webserver started at http://127.23.132.67:34657/ using document root <none> and password file <none>
I20260320 02:58:26.851514 29688 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:26.851577 29688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:26.851822 29688 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:26.853545 29688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "8983417d04774a2080494d08cc23385e"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "bf8567eccb22eb999780d883942242be"
server_key_iv: "44bf6971e40559922dff6c1619a46ee2"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.854054 29688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "8983417d04774a2080494d08cc23385e"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "bf8567eccb22eb999780d883942242be"
server_key_iv: "44bf6971e40559922dff6c1619a46ee2"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.857611 29688 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260320 02:58:26.860697 29702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.862671 29688 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:26.862819 29688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "8983417d04774a2080494d08cc23385e"
format_stamp: "Formatted at 2026-03-20 02:58:26 on dist-test-slave-v3hj"
server_key: "bf8567eccb22eb999780d883942242be"
server_key_iv: "44bf6971e40559922dff6c1619a46ee2"
server_key_version: "encryptionkey@0"
I20260320 02:58:26.862933 29688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:26.868804 29688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:26.869400 29688 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:26.869565 29688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:26.870250 29688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:26.871652 29688 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:26.871825 29688 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.871990 29688 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:26.872083 29688 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:26.885205 29688 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:46661
I20260320 02:58:26.885249 29815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:46661 every 8 connection(s)
I20260320 02:58:26.887144 29688 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:26.894037 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29688
I20260320 02:58:26.894368 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:26.895077 24081 external_mini_cluster.cc:1468] Setting key 95af4dc6e108c1b3bdaaf2a9be086894
I20260320 02:58:26.899648 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.889383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:56009 (local address 127.23.132.126:36395)
0320 02:58:26.889556 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.889561 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.890343 (+ 782us) server_negotiation.cc:408] Connection header received
0320 02:58:26.891467 (+ 1124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.891470 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.891513 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.891596 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.899019 (+ 7423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.899023 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.899026 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.899038 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.899063 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.899070 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.899254 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.899488 (+ 234us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.899532 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260320 02:58:26.900537 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.889422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:36395 (local address 127.23.132.67:56009)
0320 02:58:26.890169 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.890203 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.891042 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.891852 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.891860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.891953 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.898597 (+ 6644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.898600 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.898644 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.898754 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.899277 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.899287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.899292 (+ 5us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.899481 (+ 189us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.899679 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":132,"threads_started":1}
I20260320 02:58:26.902263 29816 heartbeater.cc:344] Connected to a master server at 127.23.132.126:36395
I20260320 02:58:26.902534 29816 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:26.903053 29816 heartbeater.cc:507] Master 127.23.132.126:36395 requested a full tablet report, sending...
I20260320 02:58:26.904386 29363 ts_manager.cc:194] Registered new tserver with Master: 8983417d04774a2080494d08cc23385e (127.23.132.67:46661)
I20260320 02:58:26.910195 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:58:26.915505 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.912371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.23.132.126:36395)
0320 02:58:26.912718 (+ 347us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.912723 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.913258 (+ 535us) server_negotiation.cc:408] Connection header received
0320 02:58:26.913308 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.913311 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.913352 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.913425 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.914548 (+ 1123us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.914553 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.914556 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.914570 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.914603 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.914612 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.914734 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.915240 (+ 506us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.915355 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260320 02:58:26.920290 29363 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:58:26.924011 29363 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:58:26.947793 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.943801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:41819 (local address 127.0.0.1:37224)
0320 02:58:26.945131 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.945144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.945284 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.946003 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.946007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.946024 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.946776 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.946778 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.946804 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.946874 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.947235 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.947244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.947249 (+ 5us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.947547 (+ 298us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.947614 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":120,"threads_started":1}
I20260320 02:58:26.948228 29827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.943740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:36313 (local address 127.0.0.1:48166)
0320 02:58:26.944345 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.944382 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.944501 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.945425 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.945434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.945523 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.946759 (+ 1236us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.946762 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.946805 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.946874 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.947977 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.947980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.947983 (+ 3us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.948070 (+ 87us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.948102 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":84,"threads_started":1}
I20260320 02:58:26.948678 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.944511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.23.132.65:36313)
0320 02:58:26.944759 (+ 248us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.944767 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.944790 (+ 23us) server_negotiation.cc:408] Connection header received
0320 02:58:26.944851 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.944855 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.945035 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.946187 (+ 1152us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.947164 (+ 977us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.947171 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.947183 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.947195 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.947244 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.947259 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.947601 (+ 342us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.948204 (+ 603us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.948371 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260320 02:58:26.948730 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.943840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.23.132.66:41819)
0320 02:58:26.945003 (+ 1163us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.945009 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.945222 (+ 213us) server_negotiation.cc:408] Connection header received
0320 02:58:26.945407 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.945413 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.945552 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.945658 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.947050 (+ 1392us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.947056 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.947064 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.947074 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.947118 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.947128 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.947258 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.948499 (+ 1241us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.948578 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":86,"threads_started":1}
I20260320 02:58:26.950456 29831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.944959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:46661 (local address 127.0.0.1:47070)
0320 02:58:26.945501 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.945518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.945610 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.948146 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.948149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.948164 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.949258 (+ 1094us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.949260 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.949273 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.949300 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.950228 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.950232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.950235 (+ 3us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.950307 (+ 72us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.950340 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":112,"threads_started":1}
I20260320 02:58:26.951808 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.945367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.23.132.67:46661)
0320 02:58:26.946374 (+ 1007us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.946380 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.946402 (+ 22us) server_negotiation.cc:408] Connection header received
0320 02:58:26.947289 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.947298 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.947591 (+ 293us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.947772 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.949957 (+ 2185us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.949963 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.949973 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.949984 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.950027 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.950037 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.950182 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.951376 (+ 1194us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.951484 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":133,"threads_started":1}
I20260320 02:58:26.952777 29619 tablet_service.cc:1511] Processing CreateTablet for tablet ee244c24eebf4fb28500ea7c44271113 (DEFAULT_TABLE table=test-table [id=419d02123607462a9f60b90ca73b7d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:26.953996 29488 tablet_service.cc:1511] Processing CreateTablet for tablet ee244c24eebf4fb28500ea7c44271113 (DEFAULT_TABLE table=test-table [id=419d02123607462a9f60b90ca73b7d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:26.955240 29488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee244c24eebf4fb28500ea7c44271113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:26.955483 29619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee244c24eebf4fb28500ea7c44271113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:26.956862 29749 tablet_service.cc:1511] Processing CreateTablet for tablet ee244c24eebf4fb28500ea7c44271113 (DEFAULT_TABLE table=test-table [id=419d02123607462a9f60b90ca73b7d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:26.957927 29749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee244c24eebf4fb28500ea7c44271113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:26.963219 29834 tablet_bootstrap.cc:492] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: Bootstrap starting.
I20260320 02:58:26.965523 29835 tablet_bootstrap.cc:492] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Bootstrap starting.
I20260320 02:58:26.965601 29833 tablet_bootstrap.cc:492] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc: Bootstrap starting.
I20260320 02:58:26.965828 29834 tablet_bootstrap.cc:654] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:26.968257 29834 log.cc:826] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:26.969266 29835 tablet_bootstrap.cc:654] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:26.969604 29833 tablet_bootstrap.cc:654] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:26.970106 29835 log.cc:826] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:26.970919 29833 log.cc:826] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:26.971194 29834 tablet_bootstrap.cc:492] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: No bootstrap required, opened a new log
I20260320 02:58:26.971472 29834 ts_tablet_manager.cc:1403] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260320 02:58:26.975582 29835 tablet_bootstrap.cc:492] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: No bootstrap required, opened a new log
I20260320 02:58:26.976054 29835 ts_tablet_manager.cc:1403] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260320 02:58:26.978791 29835 raft_consensus.cc:359] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.979007 29835 raft_consensus.cc:385] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:26.979084 29835 raft_consensus.cc:740] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983417d04774a2080494d08cc23385e, State: Initialized, Role: FOLLOWER
I20260320 02:58:26.979578 29835 consensus_queue.cc:260] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.980571 29835 ts_tablet_manager.cc:1434] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260320 02:58:26.980685 29834 raft_consensus.cc:359] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.980911 29834 raft_consensus.cc:385] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:26.980960 29834 raft_consensus.cc:740] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b8bdde5eee4d59978d0d3b81d39739, State: Initialized, Role: FOLLOWER
I20260320 02:58:26.980968 29816 heartbeater.cc:499] Master 127.23.132.126:36395 was elected leader, sending a full tablet report...
I20260320 02:58:26.981441 29834 consensus_queue.cc:260] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.982321 29685 heartbeater.cc:499] Master 127.23.132.126:36395 was elected leader, sending a full tablet report...
I20260320 02:58:26.982491 29834 ts_tablet_manager.cc:1434] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260320 02:58:26.986444 29833 tablet_bootstrap.cc:492] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc: No bootstrap required, opened a new log
I20260320 02:58:26.987130 29833 ts_tablet_manager.cc:1403] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20260320 02:58:26.987174 29840 raft_consensus.cc:493] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:26.987448 29840 raft_consensus.cc:515] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.989521 29840 leader_election.cc:290] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb756426f48140b298ef55403571f2fc (127.23.132.65:36313), 8983417d04774a2080494d08cc23385e (127.23.132.67:46661)
I20260320 02:58:26.991294 29833 raft_consensus.cc:359] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.991660 29833 raft_consensus.cc:385] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:26.991732 29833 raft_consensus.cc:740] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb756426f48140b298ef55403571f2fc, State: Initialized, Role: FOLLOWER
I20260320 02:58:26.992448 29833 consensus_queue.cc:260] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.993110 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.989408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:46661 (local address 127.23.132.66:46477)
0320 02:58:26.990128 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.990154 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.990327 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.990773 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.990777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.990794 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.992395 (+ 1601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.992397 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.992414 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.992448 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.992813 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.992818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.992822 (+ 4us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.992926 (+ 104us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.992968 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548}
I20260320 02:58:26.993724 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.989603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:46477 (local address 127.23.132.67:46661)
0320 02:58:26.989920 (+ 317us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.989926 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.990440 (+ 514us) server_negotiation.cc:408] Connection header received
0320 02:58:26.990509 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.990513 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.990567 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.990642 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.992610 (+ 1968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.992615 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.992618 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.992636 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.992665 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.992677 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.992789 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.993509 (+ 720us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.993589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260320 02:58:26.994199 29554 heartbeater.cc:499] Master 127.23.132.126:36395 was elected leader, sending a full tablet report...
I20260320 02:58:26.994378 29770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee244c24eebf4fb28500ea7c44271113" candidate_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983417d04774a2080494d08cc23385e" is_pre_election: true
I20260320 02:58:26.994683 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.990578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:34255 (local address 127.23.132.65:36313)
0320 02:58:26.990705 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:26.990709 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:26.990996 (+ 287us) server_negotiation.cc:408] Connection header received
0320 02:58:26.991196 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:26.991199 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:26.991240 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:26.991310 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:26.994084 (+ 2774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:26.994088 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:26.994091 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:26.994103 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:26.994128 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:26.994138 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:26.994195 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:26.994339 (+ 144us) server_negotiation.cc:300] Negotiation successful
0320 02:58:26.994470 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260320 02:58:26.994684 29770 raft_consensus.cc:2468] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b8bdde5eee4d59978d0d3b81d39739 in term 0.
I20260320 02:58:26.995148 29841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:26.990457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:36313 (local address 127.23.132.66:34255)
0320 02:58:26.990859 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:26.990873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:26.991068 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:26.991455 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:26.991458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:26.991477 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:26.993700 (+ 2223us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:58:26.993702 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:26.993727 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:58:26.993769 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:26.994248 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:26.994251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:26.994253 (+ 2us) client_negotiation.cc:769] Sending connection context
0320 02:58:26.994884 (+ 631us) client_negotiation.cc:240] Negotiation successful
0320 02:58:26.994924 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":112,"threads_started":1}
I20260320 02:58:26.995203 29833 ts_tablet_manager.cc:1434] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260320 02:58:26.995188 29572 leader_election.cc:304] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983417d04774a2080494d08cc23385e, b1b8bdde5eee4d59978d0d3b81d39739; no voters:
I20260320 02:58:26.995615 29840 raft_consensus.cc:2804] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:58:26.995692 29840 raft_consensus.cc:493] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:26.995800 29840 raft_consensus.cc:3060] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:26.995894 29507 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee244c24eebf4fb28500ea7c44271113" candidate_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb756426f48140b298ef55403571f2fc" is_pre_election: true
I20260320 02:58:26.996174 29507 raft_consensus.cc:2468] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b8bdde5eee4d59978d0d3b81d39739 in term 0.
I20260320 02:58:26.998701 29840 raft_consensus.cc:515] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:26.999082 29840 leader_election.cc:290] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [CANDIDATE]: Term 1 election: Requested vote from peers fb756426f48140b298ef55403571f2fc (127.23.132.65:36313), 8983417d04774a2080494d08cc23385e (127.23.132.67:46661)
I20260320 02:58:26.999624 29507 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee244c24eebf4fb28500ea7c44271113" candidate_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb756426f48140b298ef55403571f2fc"
I20260320 02:58:26.999717 29770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee244c24eebf4fb28500ea7c44271113" candidate_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983417d04774a2080494d08cc23385e"
I20260320 02:58:26.999866 29770 raft_consensus.cc:3060] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:26.999804 29507 raft_consensus.cc:3060] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:27.001711 29770 raft_consensus.cc:2468] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1b8bdde5eee4d59978d0d3b81d39739 in term 1.
I20260320 02:58:27.002177 29572 leader_election.cc:304] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8983417d04774a2080494d08cc23385e, b1b8bdde5eee4d59978d0d3b81d39739; no voters:
I20260320 02:58:27.002562 29840 raft_consensus.cc:2804] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:27.002813 29840 raft_consensus.cc:697] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 1 LEADER]: Becoming Leader. State: Replica: b1b8bdde5eee4d59978d0d3b81d39739, State: Running, Role: LEADER
I20260320 02:58:27.003207 29840 consensus_queue.cc:237] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } }
I20260320 02:58:27.003204 29507 raft_consensus.cc:2468] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1b8bdde5eee4d59978d0d3b81d39739 in term 1.
I20260320 02:58:27.011812 29366 catalog_manager.cc:5671] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1b8bdde5eee4d59978d0d3b81d39739 (127.23.132.66). New cstate: current_term: 1 leader_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b8bdde5eee4d59978d0d3b81d39739" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 41819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 } health_report { overall_health: UNKNOWN } } }
I20260320 02:58:27.038429 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:27.035855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.23.132.66:41819)
0320 02:58:27.036017 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:27.036021 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:27.036097 (+ 76us) server_negotiation.cc:408] Connection header received
0320 02:58:27.036386 (+ 289us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:27.036390 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:27.036455 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:27.036538 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:27.037889 (+ 1351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:27.037893 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:27.037896 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:27.037910 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:27.037934 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:27.037943 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:27.038126 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:27.038266 (+ 140us) server_negotiation.cc:300] Negotiation successful
0320 02:58:27.038312 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260320 02:58:27.045939 29507 raft_consensus.cc:1275] T ee244c24eebf4fb28500ea7c44271113 P fb756426f48140b298ef55403571f2fc [term 1 FOLLOWER]: Refusing update from remote peer b1b8bdde5eee4d59978d0d3b81d39739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:27.045977 29770 raft_consensus.cc:1275] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 1 FOLLOWER]: Refusing update from remote peer b1b8bdde5eee4d59978d0d3b81d39739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:27.046953 29843 consensus_queue.cc:1048] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb756426f48140b298ef55403571f2fc" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:27.047513 29840 consensus_queue.cc:1048] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983417d04774a2080494d08cc23385e" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:27.060093 29847 mvcc.cc:204] Tried to move back new op lower bound from 7266203676847656960 to 7266203676690169856. Current Snapshot: MvccSnapshot[applied={T|T < 7266203676847656960}]
I20260320 02:58:27.068102 29363 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:27.068332 29363 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:27.072489 29363 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 9bc7dc6f16a345208ae4dd9cdadacf43: Sending DeleteTablet for 3 replicas of tablet ee244c24eebf4fb28500ea7c44271113
I20260320 02:58:27.073616 29619 tablet_service.cc:1558] Processing DeleteTablet for tablet ee244c24eebf4fb28500ea7c44271113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:27 UTC) from {username='slave'} at 127.0.0.1:37224
I20260320 02:58:27.074154 29488 tablet_service.cc:1558] Processing DeleteTablet for tablet ee244c24eebf4fb28500ea7c44271113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:27 UTC) from {username='slave'} at 127.0.0.1:48166
I20260320 02:58:27.074594 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29426
I20260320 02:58:27.075930 29749 tablet_service.cc:1558] Processing DeleteTablet for tablet ee244c24eebf4fb28500ea7c44271113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:27 UTC) from {username='slave'} at 127.0.0.1:47070
I20260320 02:58:27.076555 29858 tablet_replica.cc:333] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: stopping tablet replica
I20260320 02:58:27.076934 29858 raft_consensus.cc:2243] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:27.077324 29858 raft_consensus.cc:2272] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:27.079263 29858 ts_tablet_manager.cc:1916] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:27.082250 29856 tablet_replica.cc:333] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739: stopping tablet replica
I20260320 02:58:27.082913 29858 ts_tablet_manager.cc:1929] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:27.083107 29858 log.cc:1199] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/ee244c24eebf4fb28500ea7c44271113
I20260320 02:58:27.083542 29858 ts_tablet_manager.cc:1950] T ee244c24eebf4fb28500ea7c44271113 P 8983417d04774a2080494d08cc23385e: Deleting consensus metadata
I20260320 02:58:27.082830 29856 raft_consensus.cc:2243] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:58:27.084523 29856 raft_consensus.cc:2272] T ee244c24eebf4fb28500ea7c44271113 P b1b8bdde5eee4d59978d0d3b81d39739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:27.085582 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29557
W20260320 02:58:27.085654 29353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.132.65:36313 (error 108)
I20260320 02:58:27.089504 29350 catalog_manager.cc:5002] TS 8983417d04774a2080494d08cc23385e (127.23.132.67:46661): tablet ee244c24eebf4fb28500ea7c44271113 (table test-table [id=419d02123607462a9f60b90ca73b7d69]) successfully deleted
I20260320 02:58:27.090266 29831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:27.089710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:36313 (local address 127.0.0.1:48176)
0320 02:58:27.090085 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:27.090167 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.65:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
W20260320 02:58:27.090497 29353 catalog_manager.cc:4729] TS fb756426f48140b298ef55403571f2fc (127.23.132.65:36313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee244c24eebf4fb28500ea7c44271113: Network error: Client connection negotiation failed: client connection to 127.23.132.65:36313: connect: Connection refused (error 111)
I20260320 02:58:27.094030 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29688
I20260320 02:58:27.095829 29831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:27.094522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:41819 (local address 127.0.0.1:37254)
0320 02:58:27.095667 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:27.095739 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.66:41819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1039}
I20260320 02:58:27.099880 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29335
2026-03-20T02:58:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1191 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260320 02:58:27.123781 24081 mini_oidc.cc:150] Starting JWKS server
I20260320 02:58:27.125180 24081 webserver.cc:492] Webserver started at https://127.0.0.1:38183/ using document root <none> and password file <none>
I20260320 02:58:27.125316 24081 mini_oidc.cc:186] Starting OIDC Discovery server
I20260320 02:58:27.125721 24081 webserver.cc:492] Webserver started at http://127.0.0.1:33781/ using document root <none> and password file <none>
2026-03-20T02:58:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:27Z Disabled control of system clock
I20260320 02:58:27.136781 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:44329
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:44329
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/testchainca.cert with env {}
W20260320 02:58:27.260370 29865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:27.260627 29865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:27.260679 29865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:27.264391 29865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:27.264609 29865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:27.264678 29865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:27.264701 29865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:27.264719 29865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260320 02:58:27.264737 29865 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260320 02:58:27.264756 29865 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38183/jwks.json
I20260320 02:58:27.271176 29865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:44329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:44329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:27.272351 29865 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:27.273173 29865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260320 02:58:27.281189 29865 server_base.cc:1061] running on GCE node
W20260320 02:58:27.281427 29871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:27.281220 29870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:27.281234 29873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:27.282703 29865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:27.284498 29865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:27.285862 29865 hybrid_clock.cc:648] HybridClock initialized: now 1773975507285847 us; error 32 us; skew 500 ppm
I20260320 02:58:27.287966 29865 webserver.cc:492] Webserver started at http://127.23.132.126:35177/ using document root <none> and password file <none>
I20260320 02:58:27.288528 29865 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:27.288590 29865 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:27.288770 29865 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:27.290525 29865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "0f16dd7f188b459397f2e532079e059b"
format_stamp: "Formatted at 2026-03-20 02:58:27 on dist-test-slave-v3hj"
server_key: "5126ab23d1563bc7c4de1653f0c4c292"
server_key_iv: "1bd4f6e160246b27cfa7c6d1dd8efef6"
server_key_version: "encryptionkey@0"
I20260320 02:58:27.290997 29865 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "0f16dd7f188b459397f2e532079e059b"
format_stamp: "Formatted at 2026-03-20 02:58:27 on dist-test-slave-v3hj"
server_key: "5126ab23d1563bc7c4de1653f0c4c292"
server_key_iv: "1bd4f6e160246b27cfa7c6d1dd8efef6"
server_key_version: "encryptionkey@0"
I20260320 02:58:27.294394 29865 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:27.297537 29879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:27.301393 29865 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260320 02:58:27.301692 29865 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "0f16dd7f188b459397f2e532079e059b"
format_stamp: "Formatted at 2026-03-20 02:58:27 on dist-test-slave-v3hj"
server_key: "5126ab23d1563bc7c4de1653f0c4c292"
server_key_iv: "1bd4f6e160246b27cfa7c6d1dd8efef6"
server_key_version: "encryptionkey@0"
I20260320 02:58:27.302793 29865 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:27.343092 29865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:27.343849 29865 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:27.344022 29865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:27.355298 29931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:44329 every 8 connection(s)
I20260320 02:58:27.355506 29865 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:44329
I20260320 02:58:27.356906 29865 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:27.360666 29932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:27.365458 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29865
I20260320 02:58:27.365593 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:27.366046 24081 external_mini_cluster.cc:1468] Setting key 7b0c8109fb7c11edeef43c79daeee8b8
I20260320 02:58:27.368366 29932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b: Bootstrap starting.
I20260320 02:58:27.370558 29932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:27.371235 29932 log.cc:826] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:27.374203 29932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b: No bootstrap required, opened a new log
I20260320 02:58:27.377661 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:27.368193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.23.132.126:44329)
0320 02:58:27.368830 (+ 637us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:27.368838 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:27.368871 (+ 33us) server_negotiation.cc:408] Connection header received
0320 02:58:27.369439 (+ 568us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:27.369454 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:27.369781 (+ 327us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:27.370087 (+ 306us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:27.370716 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.371937 (+ 1221us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:27.373057 (+ 1120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.373332 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:27.375358 (+ 2026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:27.375395 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:27.375407 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:27.376156 (+ 749us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:27.376332 (+ 176us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:27.376356 (+ 24us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:27.376480 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:27.376675 (+ 195us) server_negotiation.cc:300] Negotiation successful
0320 02:58:27.377054 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":190,"threads_started":1}
I20260320 02:58:27.377983 29932 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f16dd7f188b459397f2e532079e059b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44329 } }
I20260320 02:58:27.378460 29932 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:27.378532 29932 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f16dd7f188b459397f2e532079e059b, State: Initialized, Role: FOLLOWER
I20260320 02:58:27.378942 29932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f16dd7f188b459397f2e532079e059b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44329 } }
I20260320 02:58:27.379077 29932 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:27.379134 29932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:27.379444 29932 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:27.380645 29932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f16dd7f188b459397f2e532079e059b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44329 } }
I20260320 02:58:27.381377 29932 leader_election.cc:304] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f16dd7f188b459397f2e532079e059b; no voters:
I20260320 02:58:27.381976 29932 leader_election.cc:290] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:27.382108 29937 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:27.382270 29937 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [term 1 LEADER]: Becoming Leader. State: Replica: 0f16dd7f188b459397f2e532079e059b, State: Running, Role: LEADER
I20260320 02:58:27.382573 29937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f16dd7f188b459397f2e532079e059b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44329 } }
I20260320 02:58:27.383236 29932 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:27.383496 24081 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260320 02:58:27.383785 29938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f16dd7f188b459397f2e532079e059b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f16dd7f188b459397f2e532079e059b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44329 } } }
I20260320 02:58:27.383823 29939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f16dd7f188b459397f2e532079e059b. Latest consensus state: current_term: 1 leader_uuid: "0f16dd7f188b459397f2e532079e059b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f16dd7f188b459397f2e532079e059b" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 44329 } } }
I20260320 02:58:27.383922 29939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:27.384002 29938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:27.385092 29941 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:27.388021 29941 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:27.395423 29941 catalog_manager.cc:1357] Generated new cluster ID: ca633afa47c64fe79957940d12a68b78
I20260320 02:58:27.395514 29941 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:27.409137 29941 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:27.410641 29941 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:27.424621 29941 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0f16dd7f188b459397f2e532079e059b: Generated new TSK 0
I20260320 02:58:27.425393 29941 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:58:27.472039 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:27.440989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.23.132.126:44329)
0320 02:58:27.441207 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:27.441213 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:27.441495 (+ 282us) server_negotiation.cc:408] Connection header received
0320 02:58:27.441719 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:27.441723 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:27.441826 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:27.441989 (+ 163us) server_negotiation.cc:227] Negotiated authn=JWT
0320 02:58:27.442645 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.443471 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:27.444282 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.444492 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:27.444610 (+ 118us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0320 02:58:27.469730 (+ 25120us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0320 02:58:27.469907 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:27.470907 (+ 1000us) server_negotiation.cc:300] Negotiation successful
0320 02:58:27.470999 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":173,"threads_started":1}
I20260320 02:58:27.495321 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:27.483605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.23.132.126:44329)
0320 02:58:27.483775 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:27.483779 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:27.484535 (+ 756us) server_negotiation.cc:408] Connection header received
0320 02:58:27.484599 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:27.484602 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:27.484651 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:27.485968 (+ 1317us) server_negotiation.cc:227] Negotiated authn=JWT
0320 02:58:27.486366 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.487784 (+ 1418us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:27.489015 (+ 1231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.489374 (+ 359us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:27.489470 (+ 96us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0320 02:58:27.494413 (+ 4943us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0320 02:58:27.495163 (+ 750us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37454: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":86}
W20260320 02:58:27.495415 29935 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37454: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260320 02:58:27.496376 29968 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:27.483435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:44329 (local address 127.0.0.1:37454)
0320 02:58:27.483740 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:27.483755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:27.483913 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:27.485987 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:27.485990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:27.486000 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0320 02:58:27.486251 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:27.486260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.488067 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:27.488071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:27.488643 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:27.488655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:27.489035 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:27.489067 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0320 02:58:27.495104 (+ 6037us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0320 02:58:27.495299 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.132.126:44329: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
I20260320 02:58:42.506021 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:42.502142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.23.132.126:44329)
0320 02:58:42.502384 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:42.502387 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:42.502404 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:58:42.502675 (+ 271us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:42.502681 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:42.502774 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:42.502979 (+ 205us) server_negotiation.cc:227] Negotiated authn=JWT
0320 02:58:42.503407 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.504035 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:42.504832 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.505010 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:42.505065 (+ 55us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0320 02:58:42.505701 (+ 636us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0320 02:58:42.505898 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40924: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
W20260320 02:58:42.506145 29974 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40924: JWT verification failed: token expired
W20260320 02:58:42.506222 29973 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:42.501914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:44329 (local address 127.0.0.1:40924)
0320 02:58:42.502263 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:42.502276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:42.502364 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:42.502976 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:42.502979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:42.502988 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0320 02:58:42.503209 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:42.503217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.504208 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:42.504211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:42.504630 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:42.504638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.504757 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:42.504780 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0320 02:58:42.505878 (+ 1098us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0320 02:58:42.506060 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.132.126:44329: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
W20260320 02:58:42.509073 29974 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:42.508072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.23.132.126:44329)
0320 02:58:42.508274 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:42.508278 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:42.508451 (+ 173us) server_negotiation.cc:408] Connection header received
0320 02:58:42.508590 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:42.508594 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:42.508643 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:42.508709 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:42.508989 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40934: BlockingRecv error: recv got EOF from 127.0.0.1:40934 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260320 02:58:42.513171 29984 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.23.132.126:44329
W20260320 02:58:42.514047 29974 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:42.512820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.23.132.126:44329)
0320 02:58:42.513027 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:42.513031 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:42.513107 (+ 76us) server_negotiation.cc:408] Connection header received
0320 02:58:42.513342 (+ 235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:42.513345 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:42.513393 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:42.513470 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:42.513895 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40944: BlockingRecv error: recv got EOF from 127.0.0.1:40944 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260320 02:58:42.514284 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29865
2026-03-20T02:58:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15432 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260320 02:58:42.556176 24081 mini_oidc.cc:150] Starting JWKS server
I20260320 02:58:42.557175 24081 webserver.cc:492] Webserver started at https://127.0.0.1:43869/ using document root <none> and password file <none>
I20260320 02:58:42.557276 24081 mini_oidc.cc:186] Starting OIDC Discovery server
I20260320 02:58:42.557500 24081 webserver.cc:492] Webserver started at http://127.0.0.1:44367/ using document root <none> and password file <none>
2026-03-20T02:58:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:42Z Disabled control of system clock
I20260320 02:58:42.565692 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:38521
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:38521
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43869/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/testchainca.cert with env {}
W20260320 02:58:42.676151 29990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:42.676402 29990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:42.676466 29990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:42.680064 29990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:42.680150 29990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:42.680166 29990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:42.680183 29990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:42.680199 29990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260320 02:58:42.680217 29990 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260320 02:58:42.680229 29990 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43869/jwks.json
I20260320 02:58:42.684957 29990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:38521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43869/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:42.686244 29990 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:42.687170 29990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:42.694902 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:42.694962 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:42.695305 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:42.696035 29990 server_base.cc:1061] running on GCE node
I20260320 02:58:42.696612 29990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:42.697607 29990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:42.698797 29990 hybrid_clock.cc:648] HybridClock initialized: now 1773975522698741 us; error 90 us; skew 500 ppm
I20260320 02:58:42.701095 29990 webserver.cc:492] Webserver started at http://127.23.132.126:40907/ using document root <none> and password file <none>
I20260320 02:58:42.701848 29990 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:42.701921 29990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:42.702132 29990 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:42.704778 29990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "0dcb157f9fa648c4be3c65be941b0c83"
format_stamp: "Formatted at 2026-03-20 02:58:42 on dist-test-slave-v3hj"
server_key: "6c6354fb905ca529bd147b9a879ec452"
server_key_iv: "a931f6d621bfd1bc2881cea65df2ac4c"
server_key_version: "encryptionkey@0"
I20260320 02:58:42.705670 29990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "0dcb157f9fa648c4be3c65be941b0c83"
format_stamp: "Formatted at 2026-03-20 02:58:42 on dist-test-slave-v3hj"
server_key: "6c6354fb905ca529bd147b9a879ec452"
server_key_iv: "a931f6d621bfd1bc2881cea65df2ac4c"
server_key_version: "encryptionkey@0"
I20260320 02:58:42.710476 29990 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:42.713467 30004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:42.714789 29990 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260320 02:58:42.714919 29990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "0dcb157f9fa648c4be3c65be941b0c83"
format_stamp: "Formatted at 2026-03-20 02:58:42 on dist-test-slave-v3hj"
server_key: "6c6354fb905ca529bd147b9a879ec452"
server_key_iv: "a931f6d621bfd1bc2881cea65df2ac4c"
server_key_version: "encryptionkey@0"
I20260320 02:58:42.715037 29990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:42.758201 29990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:42.758863 29990 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:42.759028 29990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:42.766469 30056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:38521 every 8 connection(s)
I20260320 02:58:42.766475 29990 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:38521
I20260320 02:58:42.767550 29990 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:42.770354 30057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:42.772904 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 29990
I20260320 02:58:42.773046 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:42.773334 24081 external_mini_cluster.cc:1468] Setting key 46497ed1ba768f03973e51b0adb4ee78
I20260320 02:58:42.776396 30057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83: Bootstrap starting.
I20260320 02:58:42.778606 30057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:42.779312 30057 log.cc:826] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:42.781240 30057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83: No bootstrap required, opened a new log
I20260320 02:58:42.783665 30057 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcb157f9fa648c4be3c65be941b0c83" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38521 } }
I20260320 02:58:42.783871 30057 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:42.783941 30057 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dcb157f9fa648c4be3c65be941b0c83, State: Initialized, Role: FOLLOWER
I20260320 02:58:42.784754 30057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcb157f9fa648c4be3c65be941b0c83" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38521 } }
I20260320 02:58:42.784925 30057 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:42.784981 30057 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:42.785090 30057 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:42.786716 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:42.774961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.23.132.126:38521)
0320 02:58:42.775635 (+ 674us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:42.775646 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:42.775687 (+ 41us) server_negotiation.cc:408] Connection header received
0320 02:58:42.776256 (+ 569us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:42.776271 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:42.776634 (+ 363us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:42.776926 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:42.777483 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.778263 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:42.779365 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.779756 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:42.784594 (+ 4838us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:42.784619 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:42.784629 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:42.785206 (+ 577us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:42.785260 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:42.785277 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:42.785374 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:42.785878 (+ 504us) server_negotiation.cc:300] Negotiation successful
0320 02:58:42.786111 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
I20260320 02:58:42.786674 30057 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcb157f9fa648c4be3c65be941b0c83" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38521 } }
I20260320 02:58:42.787241 30057 leader_election.cc:304] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dcb157f9fa648c4be3c65be941b0c83; no voters:
I20260320 02:58:42.787838 30057 leader_election.cc:290] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:42.787952 30062 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:42.788304 30062 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [term 1 LEADER]: Becoming Leader. State: Replica: 0dcb157f9fa648c4be3c65be941b0c83, State: Running, Role: LEADER
I20260320 02:58:42.788627 30062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcb157f9fa648c4be3c65be941b0c83" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38521 } }
I20260320 02:58:42.789537 30057 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:42.789963 24081 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260320 02:58:42.790748 30064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dcb157f9fa648c4be3c65be941b0c83. Latest consensus state: current_term: 1 leader_uuid: "0dcb157f9fa648c4be3c65be941b0c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcb157f9fa648c4be3c65be941b0c83" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38521 } } }
I20260320 02:58:42.790876 30064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:42.791081 30063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dcb157f9fa648c4be3c65be941b0c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dcb157f9fa648c4be3c65be941b0c83" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 38521 } } }
I20260320 02:58:42.791165 30063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:42.791262 30072 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:42.797724 30072 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:42.805522 30072 catalog_manager.cc:1357] Generated new cluster ID: b1e643aad60949058c18e1082b4f211d
I20260320 02:58:42.805624 30072 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:42.841269 30072 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:42.842294 30072 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:42.852065 30072 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0dcb157f9fa648c4be3c65be941b0c83: Generated new TSK 0
I20260320 02:58:42.852823 30072 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260320 02:58:42.879825 30088 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260320 02:58:42.879967 30088 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260320 02:58:42.880618 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:42.864129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.23.132.126:38521)
0320 02:58:42.864286 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:42.864291 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:42.864325 (+ 34us) server_negotiation.cc:408] Connection header received
0320 02:58:42.864417 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:42.864421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:42.864483 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:42.864579 (+ 96us) server_negotiation.cc:227] Negotiated authn=JWT
0320 02:58:42.864980 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.865820 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:42.866516 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.866750 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:42.866862 (+ 112us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0320 02:58:42.880198 (+ 13336us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0320 02:58:42.880479 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260320 02:58:42.880698 30060 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260320 02:58:42.881227 30087 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:42.863802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:38521 (local address 127.0.0.1:60470)
0320 02:58:42.864149 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:42.864165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:42.864267 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:42.864602 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:42.864605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:42.864612 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0320 02:58:42.864834 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:42.864843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.865980 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:42.865983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:42.866411 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:42.866419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:42.866518 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:42.866540 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0320 02:58:42.880534 (+ 13994us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0320 02:58:42.880799 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.132.126:38521: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":127,"threads_started":1}
I20260320 02:58:42.882001 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 29990
2026-03-20T02:58:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (368 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260320 02:58:42.924620 24081 mini_oidc.cc:150] Starting JWKS server
I20260320 02:58:42.925531 24081 webserver.cc:492] Webserver started at https://127.0.0.1:37925/ using document root <none> and password file <none>
I20260320 02:58:42.925623 24081 mini_oidc.cc:186] Starting OIDC Discovery server
I20260320 02:58:42.926071 24081 webserver.cc:492] Webserver started at http://127.0.0.1:36339/ using document root <none> and password file <none>
2026-03-20T02:58:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:42Z Disabled control of system clock
I20260320 02:58:42.938175 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:46583
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:45685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:46583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37925/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260320 02:58:43.061197 30094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:43.061443 30094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:43.061511 30094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:43.065021 30094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:43.065105 30094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:43.065128 30094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:43.065151 30094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:43.065174 30094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260320 02:58:43.065207 30094 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260320 02:58:43.065233 30094 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37925/jwks.json
I20260320 02:58:43.071616 30094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:45685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:46583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:46583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37925/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:43.073037 30094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:43.074625 30094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:43.082166 30100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:43.082175 30102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:43.082190 30099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:43.083055 30094 server_base.cc:1061] running on GCE node
I20260320 02:58:43.083590 30094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:43.084641 30094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:43.085958 30094 hybrid_clock.cc:648] HybridClock initialized: now 1773975523085895 us; error 54 us; skew 500 ppm
I20260320 02:58:43.088788 30094 webserver.cc:492] Webserver started at http://127.23.132.126:40623/ using document root <none> and password file <none>
I20260320 02:58:43.089639 30094 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:43.089854 30094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:43.090102 30094 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:43.091904 30094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "05564f76d52241d9ba9acca0119a5da3"
format_stamp: "Formatted at 2026-03-20 02:58:43 on dist-test-slave-v3hj"
server_key: "233f36b5bc8327e9a0f3a711f11a56a4"
server_key_iv: "a0de47ff1a6b914bf44fcd5203e85c1e"
server_key_version: "encryptionkey@0"
I20260320 02:58:43.092406 30094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "05564f76d52241d9ba9acca0119a5da3"
format_stamp: "Formatted at 2026-03-20 02:58:43 on dist-test-slave-v3hj"
server_key: "233f36b5bc8327e9a0f3a711f11a56a4"
server_key_iv: "a0de47ff1a6b914bf44fcd5203e85c1e"
server_key_version: "encryptionkey@0"
I20260320 02:58:43.097345 30094 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260320 02:58:43.100464 30108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:43.102188 30094 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:43.102334 30094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "05564f76d52241d9ba9acca0119a5da3"
format_stamp: "Formatted at 2026-03-20 02:58:43 on dist-test-slave-v3hj"
server_key: "233f36b5bc8327e9a0f3a711f11a56a4"
server_key_iv: "a0de47ff1a6b914bf44fcd5203e85c1e"
server_key_version: "encryptionkey@0"
I20260320 02:58:43.102439 30094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:43.116165 30094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:43.117441 30094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:43.117651 30094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:43.128420 30161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:46583 every 8 connection(s)
I20260320 02:58:43.128752 30094 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:46583
I20260320 02:58:43.130491 30094 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:43.134459 30162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:43.135833 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30094
I20260320 02:58:43.135999 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:43.136655 24081 external_mini_cluster.cc:1468] Setting key 09151c9f96a90dc38ad98d3bdb307c8e
I20260320 02:58:43.141610 30162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3: Bootstrap starting.
I20260320 02:58:43.144096 30162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:43.145125 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:43.138202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.23.132.126:46583)
0320 02:58:43.138771 (+ 569us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:43.138780 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:43.138813 (+ 33us) server_negotiation.cc:408] Connection header received
0320 02:58:43.139331 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:43.139347 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:43.139664 (+ 317us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:43.139922 (+ 258us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:43.140522 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.141544 (+ 1022us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:43.142483 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.142825 (+ 342us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:43.143281 (+ 456us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:43.143321 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:43.143331 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:43.143687 (+ 356us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:43.143726 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:43.143740 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:43.143818 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:43.144182 (+ 364us) server_negotiation.cc:300] Negotiation successful
0320 02:58:43.144510 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260320 02:58:43.145136 30162 log.cc:826] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:43.148794 30162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3: No bootstrap required, opened a new log
I20260320 02:58:43.152333 30162 raft_consensus.cc:359] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05564f76d52241d9ba9acca0119a5da3" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46583 } }
I20260320 02:58:43.152555 30162 raft_consensus.cc:385] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:43.152593 30162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05564f76d52241d9ba9acca0119a5da3, State: Initialized, Role: FOLLOWER
I20260320 02:58:43.153050 24081 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260320 02:58:43.153236 30162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05564f76d52241d9ba9acca0119a5da3" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46583 } }
I20260320 02:58:43.153664 30162 raft_consensus.cc:399] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:43.154059 30162 raft_consensus.cc:493] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:43.154162 30162 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:43.156591 30162 raft_consensus.cc:515] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05564f76d52241d9ba9acca0119a5da3" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46583 } }
I20260320 02:58:43.157400 30162 leader_election.cc:304] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05564f76d52241d9ba9acca0119a5da3; no voters:
I20260320 02:58:43.157773 30162 leader_election.cc:290] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:43.157943 30167 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:43.158130 30167 raft_consensus.cc:697] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [term 1 LEADER]: Becoming Leader. State: Replica: 05564f76d52241d9ba9acca0119a5da3, State: Running, Role: LEADER
I20260320 02:58:43.158464 30167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05564f76d52241d9ba9acca0119a5da3" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46583 } }
I20260320 02:58:43.159422 30162 sys_catalog.cc:565] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:43.160642 30169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05564f76d52241d9ba9acca0119a5da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05564f76d52241d9ba9acca0119a5da3" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46583 } } }
I20260320 02:58:43.160779 30169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:43.160681 30168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05564f76d52241d9ba9acca0119a5da3. Latest consensus state: current_term: 1 leader_uuid: "05564f76d52241d9ba9acca0119a5da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05564f76d52241d9ba9acca0119a5da3" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46583 } } }
I20260320 02:58:43.161000 30168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3 [sys.catalog]: This master's current role is: LEADER
W20260320 02:58:43.167301 30183 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:58:43.167397 30183 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:58:43.167482 30171 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:43.168403 30171 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:43.178406 30171 catalog_manager.cc:1357] Generated new cluster ID: 39f393c83c194ff19a173128d03044be
I20260320 02:58:43.178511 30171 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:43.197011 30171 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:43.198876 30171 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:43.206175 30171 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 05564f76d52241d9ba9acca0119a5da3: Generated new TSK 0
I20260320 02:58:43.206991 30171 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260320 02:58:43.245345 30192 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260320 02:58:43.245453 30192 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260320 02:58:43.247917 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:43.218406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.23.132.126:46583)
0320 02:58:43.218529 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:43.218532 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:43.218567 (+ 35us) server_negotiation.cc:408] Connection header received
0320 02:58:43.219111 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:43.219115 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:43.219175 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:43.219265 (+ 90us) server_negotiation.cc:227] Negotiated authn=JWT
0320 02:58:43.220188 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.220727 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:43.222354 (+ 1627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.222920 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:43.223048 (+ 128us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0320 02:58:43.247219 (+ 24171us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0320 02:58:43.247733 (+ 514us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260320 02:58:43.248101 30165 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260320 02:58:43.248162 30191 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:43.218169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46583 (local address 127.0.0.1:43514)
0320 02:58:43.218490 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:43.218506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:43.218626 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:43.219300 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:43.219304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:43.219313 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0320 02:58:43.219606 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:43.219615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.220865 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:43.220869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:43.221949 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:43.221978 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.222346 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:43.222392 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0320 02:58:43.247540 (+ 25148us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0320 02:58:43.247909 (+ 369us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.132.126:46583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260320 02:58:43.249145 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30094
2026-03-20T02:58:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (367 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-03-20T02:58:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:43Z Disabled control of system clock
I20260320 02:58:43.302136 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:33673
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:33673
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/testchainca.cert with env {}
W20260320 02:58:43.429173 30196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:43.429404 30196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:43.429455 30196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:43.433043 30196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:43.433207 30196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:43.433266 30196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:43.433286 30196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:43.433307 30196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260320 02:58:43.433331 30196 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260320 02:58:43.433343 30196 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260320 02:58:43.438297 30196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:33673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:33673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:43.439356 30196 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:43.440131 30196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:43.446956 30201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:43.446949 30205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:43.446954 30203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:43.447379 30196 server_base.cc:1061] running on GCE node
I20260320 02:58:43.448283 30196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:43.449970 30196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:43.451530 30196 hybrid_clock.cc:648] HybridClock initialized: now 1773975523451405 us; error 135 us; skew 500 ppm
I20260320 02:58:43.454835 30196 webserver.cc:492] Webserver started at http://127.23.132.126:40237/ using document root <none> and password file <none>
I20260320 02:58:43.455446 30196 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:43.455509 30196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:43.456007 30196 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:43.458487 30196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "11d313bfb1794b2c8f39076337ca670c"
format_stamp: "Formatted at 2026-03-20 02:58:43 on dist-test-slave-v3hj"
server_key: "e26d23fd4d5669cfb3db28de7d4f507b"
server_key_iv: "2e9e4b6b6b42615c1ca60f559d9f6049"
server_key_version: "encryptionkey@0"
I20260320 02:58:43.459024 30196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "11d313bfb1794b2c8f39076337ca670c"
format_stamp: "Formatted at 2026-03-20 02:58:43 on dist-test-slave-v3hj"
server_key: "e26d23fd4d5669cfb3db28de7d4f507b"
server_key_iv: "2e9e4b6b6b42615c1ca60f559d9f6049"
server_key_version: "encryptionkey@0"
I20260320 02:58:43.467769 30196 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260320 02:58:43.473340 30211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:43.476375 30196 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260320 02:58:43.477108 30196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "11d313bfb1794b2c8f39076337ca670c"
format_stamp: "Formatted at 2026-03-20 02:58:43 on dist-test-slave-v3hj"
server_key: "e26d23fd4d5669cfb3db28de7d4f507b"
server_key_iv: "2e9e4b6b6b42615c1ca60f559d9f6049"
server_key_version: "encryptionkey@0"
I20260320 02:58:43.477561 30196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:43.500159 30196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:43.500815 30196 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:43.501065 30196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:43.512715 30196 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:33673
I20260320 02:58:43.512758 30263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:33673 every 8 connection(s)
I20260320 02:58:43.514922 30196 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:43.518842 30264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:43.520035 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30196
I20260320 02:58:43.520174 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:43.520440 24081 external_mini_cluster.cc:1468] Setting key c84709d7677c43e599f102f457657a51
I20260320 02:58:43.525233 30264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c: Bootstrap starting.
I20260320 02:58:43.529317 30264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:43.530220 30264 log.cc:826] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:43.531505 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:43.521859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.23.132.126:33673)
0320 02:58:43.522610 (+ 751us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:43.522618 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:43.522652 (+ 34us) server_negotiation.cc:408] Connection header received
0320 02:58:43.523247 (+ 595us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:43.523266 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:43.523684 (+ 418us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:43.524307 (+ 623us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:43.524887 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.525751 (+ 864us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:43.527379 (+ 1628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.527842 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:43.529049 (+ 1207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:43.529126 (+ 77us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:43.529148 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:58:43.530054 (+ 906us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:43.530103 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:58:43.530119 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:43.530216 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:43.530488 (+ 272us) server_negotiation.cc:300] Negotiation successful
0320 02:58:43.530903 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":226,"threads_started":1}
I20260320 02:58:43.532625 30264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c: No bootstrap required, opened a new log
I20260320 02:58:43.535539 24081 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260320 02:58:43.537783 30264 raft_consensus.cc:359] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d313bfb1794b2c8f39076337ca670c" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33673 } }
I20260320 02:58:43.538095 30264 raft_consensus.cc:385] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:43.538334 30264 raft_consensus.cc:740] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d313bfb1794b2c8f39076337ca670c, State: Initialized, Role: FOLLOWER
I20260320 02:58:43.539075 30264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d313bfb1794b2c8f39076337ca670c" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33673 } }
I20260320 02:58:43.539233 30264 raft_consensus.cc:399] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:43.539278 30264 raft_consensus.cc:493] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:43.540077 30264 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:43.541177 30264 raft_consensus.cc:515] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d313bfb1794b2c8f39076337ca670c" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33673 } }
I20260320 02:58:43.541540 30264 leader_election.cc:304] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11d313bfb1794b2c8f39076337ca670c; no voters:
I20260320 02:58:43.541937 30264 leader_election.cc:290] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:43.542268 30270 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:43.542443 30270 raft_consensus.cc:697] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [term 1 LEADER]: Becoming Leader. State: Replica: 11d313bfb1794b2c8f39076337ca670c, State: Running, Role: LEADER
I20260320 02:58:43.542738 30270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d313bfb1794b2c8f39076337ca670c" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33673 } }
I20260320 02:58:43.543213 30264 sys_catalog.cc:565] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:43.545634 30271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11d313bfb1794b2c8f39076337ca670c. Latest consensus state: current_term: 1 leader_uuid: "11d313bfb1794b2c8f39076337ca670c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d313bfb1794b2c8f39076337ca670c" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33673 } } }
I20260320 02:58:43.545657 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11d313bfb1794b2c8f39076337ca670c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d313bfb1794b2c8f39076337ca670c" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33673 } } }
I20260320 02:58:43.545857 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:43.545857 30271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:43.546892 30278 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:43.550174 30278 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:43.556716 30278 catalog_manager.cc:1357] Generated new cluster ID: 7a82d87c04744f609969f1d15be20b5a
I20260320 02:58:43.556864 30278 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:43.577416 30278 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:43.578749 30278 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:43.588789 30278 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 11d313bfb1794b2c8f39076337ca670c: Generated new TSK 0
I20260320 02:58:43.589619 30278 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:58:43.785132 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:43.607680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.23.132.126:33673)
0320 02:58:43.607861 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:43.607865 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:43.607949 (+ 84us) server_negotiation.cc:408] Connection header received
0320 02:58:43.608250 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:43.608254 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:43.608341 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:43.608471 (+ 130us) server_negotiation.cc:227] Negotiated authn=JWT
0320 02:58:43.609144 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.610335 (+ 1191us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:43.611365 (+ 1030us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.611533 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:43.611634 (+ 101us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0320 02:58:43.784570 (+172936us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0320 02:58:43.784967 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36154: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":90}
W20260320 02:58:43.785187 30293 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:43.607266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:33673 (local address 127.0.0.1:36154)
0320 02:58:43.607789 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:43.607810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:43.608045 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:43.608562 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:43.608567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:43.608580 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0320 02:58:43.608998 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:43.609010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.610488 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:43.610492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:43.611200 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:43.611212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:43.611347 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:43.611376 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0320 02:58:43.784833 (+173457us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0320 02:58:43.785006 (+ 173us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.132.126:33673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
W20260320 02:58:43.785226 30267 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36154: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20260320 02:58:43.786206 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30196
2026-03-20T02:58:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (516 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (143 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (147 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:44 dist-test-slave-v3hj krb5kdc[30305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:44 dist-test-slave-v3hj krb5kdc[30305](info): set up 2 sockets
Mar 20 02:58:44 dist-test-slave-v3hj krb5kdc[30305](info): commencing operation
krb5kdc: starting...
W20260320 02:58:46.307705 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.207s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:46 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:46.514940 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:33371
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:33371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:46.661384 30321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:46.661864 30321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:46.661981 30321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:46.665769 30321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:46.665874 30321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:46.665899 30321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:46.665915 30321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:46.665934 30321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:46.672353 30321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:33371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:46.674094 30321 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:46.675216 30321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:46.682803 30326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:46.682794 30327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:46.682794 30329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:46.683166 30321 server_base.cc:1061] running on GCE node
I20260320 02:58:46.683895 30321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:46.685489 30321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:46.686822 30321 hybrid_clock.cc:648] HybridClock initialized: now 1773975526686748 us; error 98 us; skew 500 ppm
Mar 20 02:58:46 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975526, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:46.690274 30321 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:58:46.691468 30321 webserver.cc:492] Webserver started at http://127.23.132.126:39967/ using document root <none> and password file <none>
I20260320 02:58:46.692235 30321 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:46.692301 30321 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:46.692565 30321 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:46.694528 30321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "9d61d914195d4bca8ea0e0f0651fbcfe"
format_stamp: "Formatted at 2026-03-20 02:58:46 on dist-test-slave-v3hj"
server_key: "51ee27db93d0ace9f49ff480484684ea"
server_key_iv: "e0aeba55c4284966938c7d28def3d14d"
server_key_version: "encryptionkey@0"
I20260320 02:58:46.694994 30321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "9d61d914195d4bca8ea0e0f0651fbcfe"
format_stamp: "Formatted at 2026-03-20 02:58:46 on dist-test-slave-v3hj"
server_key: "51ee27db93d0ace9f49ff480484684ea"
server_key_iv: "e0aeba55c4284966938c7d28def3d14d"
server_key_version: "encryptionkey@0"
I20260320 02:58:46.699183 30321 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:46.702076 30336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:46.703430 30321 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:46.703564 30321 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "9d61d914195d4bca8ea0e0f0651fbcfe"
format_stamp: "Formatted at 2026-03-20 02:58:46 on dist-test-slave-v3hj"
server_key: "51ee27db93d0ace9f49ff480484684ea"
server_key_iv: "e0aeba55c4284966938c7d28def3d14d"
server_key_version: "encryptionkey@0"
I20260320 02:58:46.703657 30321 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:46.720896 30321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:46.724020 30321 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:46.724189 30321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:46.737186 30321 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:33371
I20260320 02:58:46.737198 30388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:33371 every 8 connection(s)
I20260320 02:58:46.738752 30321 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:46.742117 30389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:46.745857 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30321
I20260320 02:58:46.746017 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:46.746259 24081 external_mini_cluster.cc:1468] Setting key 7bc40df1b9fa86c3deb5deaa626caec0
I20260320 02:58:46.748453 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe: Bootstrap starting.
I20260320 02:58:46.751142 30389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:46.751837 30389 log.cc:826] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:46.755440 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe: No bootstrap required, opened a new log
Mar 20 02:58:46 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:46.759974 30389 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33371 } }
I20260320 02:58:46.760185 30389 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:46.760244 30389 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d61d914195d4bca8ea0e0f0651fbcfe, State: Initialized, Role: FOLLOWER
I20260320 02:58:46.760733 30389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33371 } }
I20260320 02:58:46.760883 30389 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:46.760952 30389 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:46.761234 30389 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:46.762486 30389 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33371 } }
I20260320 02:58:46.763420 30389 leader_election.cc:304] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d61d914195d4bca8ea0e0f0651fbcfe; no voters:
I20260320 02:58:46.764067 30389 leader_election.cc:290] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:46.764437 30394 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:46.764676 30394 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [term 1 LEADER]: Becoming Leader. State: Replica: 9d61d914195d4bca8ea0e0f0651fbcfe, State: Running, Role: LEADER
I20260320 02:58:46.765017 30394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33371 } }
I20260320 02:58:46.765585 30389 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:46.766758 30396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d61d914195d4bca8ea0e0f0651fbcfe. Latest consensus state: current_term: 1 leader_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33371 } } }
I20260320 02:58:46.766880 30396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:46.767246 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:46.747949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.23.132.126:33371)
0320 02:58:46.748888 (+ 939us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:46.748899 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:46.748939 (+ 40us) server_negotiation.cc:408] Connection header received
0320 02:58:46.749663 (+ 724us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:46.749688 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:46.750109 (+ 421us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:46.750452 (+ 343us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:46.752552 (+ 2100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:46.753417 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:46.754703 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:46.754940 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:46.759449 (+ 4509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:46.759478 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:46.759495 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:46.759547 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:46.762864 (+ 3317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:46.763836 (+ 972us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:46.763843 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:46.763848 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:46.763946 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:46.764314 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:46.764319 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:46.764321 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:46.764875 (+ 554us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:46.765070 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:46.766134 (+ 1064us) server_negotiation.cc:300] Negotiation successful
0320 02:58:46.766450 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":280,"threads_started":1}
I20260320 02:58:46.767453 30402 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:46.767256 30395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d61d914195d4bca8ea0e0f0651fbcfe" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 33371 } } }
I20260320 02:58:46.768468 30395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:46.772606 30402 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:46.780697 30402 catalog_manager.cc:1357] Generated new cluster ID: e994f63db36f4ce891f5162735016cd9
I20260320 02:58:46.780797 30402 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:46.792814 30402 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:46.793771 30402 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:46.802484 30402 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9d61d914195d4bca8ea0e0f0651fbcfe: Generated new TSK 0
I20260320 02:58:46.803813 30402 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:46.892426 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:33371
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:47.008936 30417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:47.009169 30417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:47.009234 30417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:47.013284 30417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:47.013386 30417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:47.013477 30417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:47.020473 30417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:33371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:47.021719 30417 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:47.022598 30417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:47.030391 30425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:47.030776 30422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:47.030438 30423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:47.031544 30417 server_base.cc:1061] running on GCE node
I20260320 02:58:47.032442 30417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:47.033164 30417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:47.034345 30417 hybrid_clock.cc:648] HybridClock initialized: now 1773975527034308 us; error 57 us; skew 500 ppm
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.041059 30417 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:58:47.042755 30417 webserver.cc:492] Webserver started at http://127.23.132.65:44311/ using document root <none> and password file <none>
I20260320 02:58:47.043372 30417 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:47.043426 30417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:47.043607 30417 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:47.045586 30417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "f3553373822949a2b118ed16a7fdfcb1"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "5203df88e68240325b974ec11deb0e3b"
server_key_iv: "75c36afe23930d2e824e4cd41d3213ce"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.046147 30417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "f3553373822949a2b118ed16a7fdfcb1"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "5203df88e68240325b974ec11deb0e3b"
server_key_iv: "75c36afe23930d2e824e4cd41d3213ce"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.050602 30417 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:58:47.053499 30432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.054698 30417 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260320 02:58:47.054878 30417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "f3553373822949a2b118ed16a7fdfcb1"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "5203df88e68240325b974ec11deb0e3b"
server_key_iv: "75c36afe23930d2e824e4cd41d3213ce"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.055001 30417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:47.078794 30417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:47.083178 30417 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:47.083375 30417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:47.084148 30417 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:47.085340 30417 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:47.085527 30417 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.085678 30417 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:47.085856 30417 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.103600 30417 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:46019
I20260320 02:58:47.103765 30545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:46019 every 8 connection(s)
I20260320 02:58:47.105927 30417 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:47.112421 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30417
I20260320 02:58:47.112718 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:47.113209 24081 external_mini_cluster.cc:1468] Setting key 7829f5a2cca86a1871bd64eb37c12411
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:47.128661 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.110063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:47689 (local address 127.23.132.126:33371)
0320 02:58:47.110490 (+ 427us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.110495 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.112275 (+ 1780us) server_negotiation.cc:408] Connection header received
0320 02:58:47.114142 (+ 1867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.114146 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.114208 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.114318 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.117749 (+ 3431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.118627 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.119588 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.119781 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.123837 (+ 4056us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:47.123862 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:47.123864 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:47.123896 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:47.125492 (+ 1596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.126263 (+ 771us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.126267 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.126269 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.126350 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.126975 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.126980 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.126982 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.127212 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:47.127315 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:47.128317 (+ 1002us) server_negotiation.cc:300] Negotiation successful
0320 02:58:47.128484 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260320 02:58:47.130573 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.110438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:33371 (local address 127.23.132.65:47689)
0320 02:58:47.111912 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:47.112116 (+ 204us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:47.113705 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:47.114611 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:47.114643 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:47.115439 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:47.117487 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:47.117518 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.118812 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.118816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:47.119425 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:47.119438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.119714 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.120486 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:47.120612 (+ 126us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:47.123556 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:47.125637 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:47.125644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:47.125654 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:47.126085 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:47.126495 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:47.126500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:47.126502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:47.126683 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:47.127395 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:47.127402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:47.127961 (+ 559us) client_negotiation.cc:769] Sending connection context
0320 02:58:47.129156 (+ 1195us) client_negotiation.cc:240] Negotiation successful
0320 02:58:47.129504 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":237,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:58:47.133500 30546 heartbeater.cc:344] Connected to a master server at 127.23.132.126:33371
I20260320 02:58:47.134615 30546 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:47.135648 30546 heartbeater.cc:507] Master 127.23.132.126:33371 requested a full tablet report, sending...
I20260320 02:58:47.138141 30352 ts_manager.cc:194] Registered new tserver with Master: f3553373822949a2b118ed16a7fdfcb1 (127.23.132.65:46019)
I20260320 02:58:47.139784 30352 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:47689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:47.200359 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:33371
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:47.319702 30553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:47.319928 30553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:47.320013 30553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:47.323343 30553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:47.323418 30553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:47.323531 30553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:47.328152 30553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:33371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:47.329630 30553 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:47.330585 30553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:47.338225 30561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:47.338286 30558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:47.338387 30559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:47.338953 30553 server_base.cc:1061] running on GCE node
I20260320 02:58:47.339459 30553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:47.340071 30553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:47.341240 30553 hybrid_clock.cc:648] HybridClock initialized: now 1773975527341218 us; error 39 us; skew 500 ppm
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.344043 30553 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:58:47.345163 30553 webserver.cc:492] Webserver started at http://127.23.132.66:34953/ using document root <none> and password file <none>
I20260320 02:58:47.345741 30553 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:47.345827 30553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:47.346015 30553 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:47.347882 30553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "ef86e3a0c25f4ca1a8f3d44ecbb44778"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "34d3a6e720eeb808c8f376de2704e3f3"
server_key_iv: "07bc94bd9426a0cffdab381d615d3e3f"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.348696 30553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "ef86e3a0c25f4ca1a8f3d44ecbb44778"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "34d3a6e720eeb808c8f376de2704e3f3"
server_key_iv: "07bc94bd9426a0cffdab381d615d3e3f"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.352233 30553 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260320 02:58:47.355139 30568 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.356585 30553 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:47.356824 30553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "ef86e3a0c25f4ca1a8f3d44ecbb44778"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "34d3a6e720eeb808c8f376de2704e3f3"
server_key_iv: "07bc94bd9426a0cffdab381d615d3e3f"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.357061 30553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:47.369856 30553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:47.375260 30553 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:47.375504 30553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:47.376205 30553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:47.377233 30553 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:47.377283 30553 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.377337 30553 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:47.377367 30553 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.392377 30553 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:35591
I20260320 02:58:47.392419 30681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:35591 every 8 connection(s)
I20260320 02:58:47.393334 30553 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:47.398414 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30553
I20260320 02:58:47.398533 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:47.398759 24081 external_mini_cluster.cc:1468] Setting key 1ef98ccd0ac49222e2d95cf40d2ec9d9
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:47.409374 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.395740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:54519 (local address 127.23.132.126:33371)
0320 02:58:47.396010 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.396014 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.396577 (+ 563us) server_negotiation.cc:408] Connection header received
0320 02:58:47.397447 (+ 870us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.397451 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.397510 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.397587 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.399368 (+ 1781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.400050 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.401214 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.401446 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.404567 (+ 3121us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:47.404592 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:47.404595 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:47.404629 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:47.406718 (+ 2089us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.407333 (+ 615us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.407337 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.407339 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.407397 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.408080 (+ 683us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.408084 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.408086 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.408308 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:47.408417 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:47.409063 (+ 646us) server_negotiation.cc:300] Negotiation successful
0320 02:58:47.409203 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:58:47.410076 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.395994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:33371 (local address 127.23.132.66:54519)
0320 02:58:47.396439 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:47.396471 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:47.397263 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:47.397743 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:47.397751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:47.398211 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:47.399181 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:47.399200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.400283 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.400290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:47.401028 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:47.401043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.401234 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.402332 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:47.402359 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:47.404279 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:47.406935 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:47.406942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:47.406954 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:47.407218 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:47.407554 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:47.407559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:47.407561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:47.407971 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:47.408452 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:47.408459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:47.408791 (+ 332us) client_negotiation.cc:769] Sending connection context
0320 02:58:47.408992 (+ 201us) client_negotiation.cc:240] Negotiation successful
0320 02:58:47.409233 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260320 02:58:47.411825 30682 heartbeater.cc:344] Connected to a master server at 127.23.132.126:33371
I20260320 02:58:47.412132 30682 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
I20260320 02:58:47.413312 30682 heartbeater.cc:507] Master 127.23.132.126:33371 requested a full tablet report, sending...
I20260320 02:58:47.414628 30352 ts_manager.cc:194] Registered new tserver with Master: ef86e3a0c25f4ca1a8f3d44ecbb44778 (127.23.132.66:35591)
I20260320 02:58:47.415225 30352 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:54519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:58:47.473421 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:33371
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:47.609460 30689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:47.610064 30689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:47.610348 30689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:47.614342 30689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:47.614421 30689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:47.614562 30689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:47.620250 30689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:33371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:47.621382 30689 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:47.622295 30689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:47.631263 30694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:47.631476 30695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:47.631641 30697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:47.632566 30689 server_base.cc:1061] running on GCE node
I20260320 02:58:47.633162 30689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:47.634018 30689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:47.635224 30689 hybrid_clock.cc:648] HybridClock initialized: now 1773975527635214 us; error 42 us; skew 500 ppm
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.639232 30689 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:58:47.640564 30689 webserver.cc:492] Webserver started at http://127.23.132.67:46615/ using document root <none> and password file <none>
I20260320 02:58:47.641134 30689 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:47.641186 30689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:47.641367 30689 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:47.643146 30689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "4342f0455dea4ac69e78f06d4516b826"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "a2e80ab64c07fa700c7c316d08a09db3"
server_key_iv: "200ee197265c435a229073ed53b740a6"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.643620 30689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "4342f0455dea4ac69e78f06d4516b826"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "a2e80ab64c07fa700c7c316d08a09db3"
server_key_iv: "200ee197265c435a229073ed53b740a6"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.647846 30689 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:58:47.650519 30704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.652858 30689 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:47.653072 30689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "4342f0455dea4ac69e78f06d4516b826"
format_stamp: "Formatted at 2026-03-20 02:58:47 on dist-test-slave-v3hj"
server_key: "a2e80ab64c07fa700c7c316d08a09db3"
server_key_iv: "200ee197265c435a229073ed53b740a6"
server_key_version: "encryptionkey@0"
I20260320 02:58:47.653411 30689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:47.669158 30689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:47.673307 30689 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:47.673602 30689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:47.674513 30689 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:47.676649 30689 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:47.676749 30689 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.676795 30689 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:47.676808 30689 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:47.689045 30689 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:35719
I20260320 02:58:47.689148 30817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:35719 every 8 connection(s)
I20260320 02:58:47.690074 30689 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:47.691577 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30689
I20260320 02:58:47.691725 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:47.692080 24081 external_mini_cluster.cc:1468] Setting key 88c2209c662dd05a26561b47228ab799
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:58:47.708770 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.692657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:33815 (local address 127.23.132.126:33371)
0320 02:58:47.692897 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.692902 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.694074 (+ 1172us) server_negotiation.cc:408] Connection header received
0320 02:58:47.695545 (+ 1471us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.695548 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.695611 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.695686 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.697405 (+ 1719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.698125 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.698995 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.699191 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.703232 (+ 4041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:47.703260 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:47.703263 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:47.703300 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:47.705002 (+ 1702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.706275 (+ 1273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.706279 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.706281 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.706344 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.707189 (+ 845us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.707193 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.707195 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.707382 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:47.707465 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:47.708384 (+ 919us) server_negotiation.cc:300] Negotiation successful
0320 02:58:47.708564 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260320 02:58:47.709689 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.692891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:33371 (local address 127.23.132.67:33815)
0320 02:58:47.693870 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:47.693925 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:47.695348 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:47.695874 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:47.695883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:47.696307 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:47.697254 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:47.697269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.698339 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.698344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:47.698841 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:47.698852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.699150 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.700062 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:47.700103 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:47.702828 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:47.705373 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:47.705383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:47.705403 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:47.706121 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:47.706544 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:47.706555 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:47.706557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:47.706996 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:47.707554 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:47.707563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:47.707990 (+ 427us) client_negotiation.cc:769] Sending connection context
0320 02:58:47.708312 (+ 322us) client_negotiation.cc:240] Negotiation successful
0320 02:58:47.708726 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":187,"threads_started":1}
I20260320 02:58:47.711212 30818 heartbeater.cc:344] Connected to a master server at 127.23.132.126:33371
I20260320 02:58:47.711580 30818 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:47.712357 30818 heartbeater.cc:507] Master 127.23.132.126:33371 requested a full tablet report, sending...
I20260320 02:58:47.713735 30352 ts_manager.cc:194] Registered new tserver with Master: 4342f0455dea4ac69e78f06d4516b826 (127.23.132.67:35719)
I20260320 02:58:47.714388 30352 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:33815
I20260320 02:58:47.720450 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/test-admin.keytab.
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.741859 24081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260320 02:58:47.745009 30825 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:47.743575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.23.132.65:46019)
0320 02:58:47.743858 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.743863 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.743884 (+ 21us) server_negotiation.cc:408] Connection header received
0320 02:58:47.743937 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.743941 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.744094 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.744206 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.744854 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60562: BlockingRecv error: recv got EOF from 127.0.0.1:60562 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
W20260320 02:58:47.745030 24081 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc' of type FILE
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.746534 24081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.747676 24081 init.cc:295] Successfully reacquired a new kerberos TGT
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:58:47.762048 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.750026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.23.132.65:46019)
0320 02:58:47.750265 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.750270 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.750290 (+ 20us) server_negotiation.cc:408] Connection header received
0320 02:58:47.750349 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.750353 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.750428 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.750532 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.751942 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.752687 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.753665 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.753943 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.756705 (+ 2762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:47.756751 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:47.756765 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:47.756809 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:47.759525 (+ 2716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.759996 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.760002 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.760003 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.760078 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.760448 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.760451 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.760452 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.760724 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:47.760928 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:47.761393 (+ 465us) server_negotiation.cc:300] Negotiation successful
0320 02:58:47.761652 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260320 02:58:47.767096 30825 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:47.765781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.23.132.65:46019)
0320 02:58:47.765946 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.765950 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.765967 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:58:47.766162 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.766165 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.766219 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.766298 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.766960 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60580: BlockingRecv error: recv got EOF from 127.0.0.1:60580 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260320 02:58:47.767323 24081 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc)
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.768620 24081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.769577 24081 init.cc:295] Successfully reacquired a new kerberos TGT
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:58:47.780367 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.770678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.23.132.65:46019)
0320 02:58:47.771112 (+ 434us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.771168 (+ 56us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.771186 (+ 18us) server_negotiation.cc:408] Connection header received
0320 02:58:47.771226 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.771229 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.771272 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.771351 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.772101 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.772664 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.773740 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.773987 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.776013 (+ 2026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:47.776043 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:47.776047 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:47.776092 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:47.778062 (+ 1970us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.778613 (+ 551us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.778617 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.778618 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.778675 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.779213 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.779216 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.779217 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.779405 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:47.779540 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:47.779901 (+ 361us) server_negotiation.cc:300] Negotiation successful
0320 02:58:47.780061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260320 02:58:47.783809 30825 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:58:47.782804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.23.132.65:46019)
0320 02:58:47.783002 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.783007 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.783061 (+ 54us) server_negotiation.cc:408] Connection header received
0320 02:58:47.783193 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.783196 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.783238 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.783300 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.783695 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60588: BlockingRecv error: recv got EOF from 127.0.0.1:60588 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260320 02:58:47.784066 24081 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773975462196075-24081-0/krb5kdc/krb5cc' of type FILE
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.785372 24081 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:47.786525 24081 init.cc:295] Successfully reacquired a new kerberos TGT
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.65@KRBTEST.COM
I20260320 02:58:47.799516 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:47.787625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.23.132.65:46019)
0320 02:58:47.787740 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:47.787743 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:47.787948 (+ 205us) server_negotiation.cc:408] Connection header received
0320 02:58:47.788070 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:47.788074 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:47.788117 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:47.788181 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:47.788786 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.789252 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:47.789858 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:47.790029 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:47.795987 (+ 5958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:47.796034 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:47.796038 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:47.796089 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:47.797781 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.798299 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.798303 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.798305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.798353 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:47.798555 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:47.798558 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:47.798560 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:47.798744 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:47.798898 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:47.799209 (+ 311us) server_negotiation.cc:300] Negotiation successful
0320 02:58:47.799329 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260320 02:58:47.801244 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30417
I20260320 02:58:47.807595 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30553
I20260320 02:58:47.814761 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30689
I20260320 02:58:47.821969 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30321
2026-03-20T02:58:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3743 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30838](info): set up 2 sockets
Mar 20 02:58:47 dist-test-slave-v3hj krb5kdc[30838](info): commencing operation
krb5kdc: starting...
W20260320 02:58:49.865002 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.021s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:58:49 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:58:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:58:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:58:50.046134 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:42639
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:50.175133 30854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:50.175374 30854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:50.175429 30854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:50.179088 30854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:50.179359 30854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:50.179442 30854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:50.179479 30854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:50.179507 30854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:50.186033 30854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:42639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:50.187597 30854 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:50.188470 30854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:50.198629 30862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:50.198791 30860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:50.198599 30859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:50.201529 30854 server_base.cc:1061] running on GCE node
I20260320 02:58:50.202606 30854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:50.204466 30854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:50.205937 30854 hybrid_clock.cc:648] HybridClock initialized: now 1773975530206006 us; error 131 us; skew 500 ppm
Mar 20 02:58:50 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:50.210108 30854 init.cc:377] Logged in from keytab as oryx/127.23.132.126@KRBTEST.COM (short username oryx)
I20260320 02:58:50.211920 30854 webserver.cc:492] Webserver started at http://127.23.132.126:41275/ using document root <none> and password file <none>
I20260320 02:58:50.212507 30854 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:50.212795 30854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:50.213030 30854 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:50.216030 30854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "a391b242632f4333b94724207e5d6889"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "4426ab098cab12e492aa6235917122bf"
server_key_iv: "26daebbe31c9fce1805fafbd54ec25e3"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.217306 30854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "a391b242632f4333b94724207e5d6889"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "4426ab098cab12e492aa6235917122bf"
server_key_iv: "26daebbe31c9fce1805fafbd54ec25e3"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.222134 30854 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260320 02:58:50.224586 30869 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.226977 30854 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:50.227335 30854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "a391b242632f4333b94724207e5d6889"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "4426ab098cab12e492aa6235917122bf"
server_key_iv: "26daebbe31c9fce1805fafbd54ec25e3"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.227478 30854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:50.249410 30854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:50.254731 30854 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:50.255378 30854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:50.267011 30854 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:42639
I20260320 02:58:50.267087 30921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:42639 every 8 connection(s)
I20260320 02:58:50.268846 30854 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:58:50.272603 30922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:50.275980 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30854
I20260320 02:58:50.276110 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:58:50.276332 24081 external_mini_cluster.cc:1468] Setting key 6e0c8123a68138ceb880481fbb5b0895
I20260320 02:58:50.278821 30922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Bootstrap starting.
I20260320 02:58:50.281219 30922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:50.282222 30922 log.cc:826] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:50.284979 30922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: No bootstrap required, opened a new log
Mar 20 02:58:50 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:58:50.288570 30922 raft_consensus.cc:359] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } }
I20260320 02:58:50.288794 30922 raft_consensus.cc:385] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:50.288869 30922 raft_consensus.cc:740] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391b242632f4333b94724207e5d6889, State: Initialized, Role: FOLLOWER
I20260320 02:58:50.289559 30922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } }
I20260320 02:58:50.289817 30922 raft_consensus.cc:399] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:58:50.289891 30922 raft_consensus.cc:493] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:58:50.290055 30922 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:50.291730 30922 raft_consensus.cc:515] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } }
I20260320 02:58:50.292315 30922 leader_election.cc:304] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a391b242632f4333b94724207e5d6889; no voters:
I20260320 02:58:50.292649 30922 leader_election.cc:290] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:58:50.292757 30927 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:50.293007 30927 raft_consensus.cc:697] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 LEADER]: Becoming Leader. State: Replica: a391b242632f4333b94724207e5d6889, State: Running, Role: LEADER
I20260320 02:58:50.293380 30927 consensus_queue.cc:237] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } }
I20260320 02:58:50.293462 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:50.278183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.23.132.126:42639)
0320 02:58:50.278721 (+ 538us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:50.278732 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:50.278817 (+ 85us) server_negotiation.cc:408] Connection header received
0320 02:58:50.279515 (+ 698us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:50.279532 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:50.279828 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:50.280102 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:50.281319 (+ 1217us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.282248 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:50.283072 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.283344 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:50.286482 (+ 3138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:50.286503 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:50.286516 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:50.286555 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:50.289130 (+ 2575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:50.289572 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:50.289577 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:50.289582 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:50.289663 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:50.290078 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:50.290083 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:50.290085 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:50.290509 (+ 424us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:50.290729 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:50.291315 (+ 586us) server_negotiation.cc:300] Negotiation successful
0320 02:58:50.291982 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":241,"threads_started":1}
I20260320 02:58:50.294024 30922 sys_catalog.cc:565] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:50.295113 30929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } }
I20260320 02:58:50.295274 30929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:50.295789 30928 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a391b242632f4333b94724207e5d6889. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } }
I20260320 02:58:50.296048 30928 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:50.296531 30936 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:50.300323 30936 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:50.309535 30936 catalog_manager.cc:1357] Generated new cluster ID: 0460188e15544f349f52287fff0d88cd
I20260320 02:58:50.309724 30936 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:50.332232 30936 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:58:50.333460 30936 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:50.340946 30936 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Generated new TSK 0
I20260320 02:58:50.342226 30936 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:58:50.426860 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:42639
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:50.552721 30950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:50.552976 30950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:50.553035 30950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:50.556793 30950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:50.556926 30950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:50.557034 30950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:50.562988 30950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:50.564167 30950 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:50.565006 30950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:50.572547 30956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:50.572669 30958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:50.572543 30955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:50.572634 30950 server_base.cc:1061] running on GCE node
I20260320 02:58:50.573522 30950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:50.574434 30950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:50.575645 30950 hybrid_clock.cc:648] HybridClock initialized: now 1773975530575599 us; error 74 us; skew 500 ppm
Mar 20 02:58:50 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:50.578779 30950 init.cc:377] Logged in from keytab as oryx/127.23.132.65@KRBTEST.COM (short username oryx)
I20260320 02:58:50.579950 30950 webserver.cc:492] Webserver started at http://127.23.132.65:35289/ using document root <none> and password file <none>
I20260320 02:58:50.580547 30950 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:50.580611 30950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:50.580775 30950 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:50.583009 30950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "3afdd0e581c244e6a0a63ca52c94a592"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "e84e58655969fdf79bad76f353d31494"
server_key_iv: "b16b1d49af832e1cc49501000f92d014"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.583484 30950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "3afdd0e581c244e6a0a63ca52c94a592"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "e84e58655969fdf79bad76f353d31494"
server_key_iv: "b16b1d49af832e1cc49501000f92d014"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.587513 30950 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:58:50.590314 30965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.591451 30950 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:50.591558 30950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "3afdd0e581c244e6a0a63ca52c94a592"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "e84e58655969fdf79bad76f353d31494"
server_key_iv: "b16b1d49af832e1cc49501000f92d014"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.591655 30950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:50.603976 30950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:50.607545 30950 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:50.607755 30950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:50.608803 30950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:50.610904 30950 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:50.610967 30950 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.611021 30950 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:50.611044 30950 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.625347 30950 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:40327
I20260320 02:58:50.625362 31078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:40327 every 8 connection(s)
I20260320 02:58:50.626401 30950 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:50.636572 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 30950
I20260320 02:58:50.636772 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:58:50.637060 24081 external_mini_cluster.cc:1468] Setting key c264724f7343d7ddb1875cd979f93ebe
Mar 20 02:58:50 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:58:50.642619 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:50.628504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:44721 (local address 127.23.132.126:42639)
0320 02:58:50.628674 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:50.628678 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:50.629995 (+ 1317us) server_negotiation.cc:408] Connection header received
0320 02:58:50.631653 (+ 1658us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:50.631658 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:50.631744 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:50.631936 (+ 192us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:50.633452 (+ 1516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.634034 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:50.634678 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.634851 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:50.638617 (+ 3766us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:50.638637 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:50.638640 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:50.638730 (+ 90us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:50.640360 (+ 1630us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:50.640998 (+ 638us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:50.641002 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:50.641004 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:50.641058 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:50.641343 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:50.641347 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:50.641348 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:50.641532 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:50.641618 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:50.642328 (+ 710us) server_negotiation.cc:300] Negotiation successful
0320 02:58:50.642448 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260320 02:58:50.643461 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:50.629047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:44721)
0320 02:58:50.629717 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:50.629758 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:50.631184 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:50.632218 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:50.632228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:50.632672 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:50.633273 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:50.633299 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.634155 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:50.634159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:50.634539 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:50.634545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.634751 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:50.636542 (+ 1791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:50.636569 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:50.638396 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:50.640513 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:50.640522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:50.640537 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:50.640892 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:50.641159 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:50.641162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:50.641163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:50.641259 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:50.641686 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:50.641693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:50.642033 (+ 340us) client_negotiation.cc:769] Sending connection context
0320 02:58:50.642274 (+ 241us) client_negotiation.cc:240] Negotiation successful
0320 02:58:50.642576 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":183,"threads_started":1}
I20260320 02:58:50.644816 31079 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42639
I20260320 02:58:50.645061 31079 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:50.645701 31079 heartbeater.cc:507] Master 127.23.132.126:42639 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:58:50.647630 30886 ts_manager.cc:194] Registered new tserver with Master: 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327)
I20260320 02:58:50.649592 30886 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:44721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:58:50.703797 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:42639
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:50.845669 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:50.845996 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:50.846076 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:50.849579 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:50.849663 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:50.849773 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:50.854817 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:50.856499 31086 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:50.857380 31086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:50.864787 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:50.864821 31094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:50.864820 31091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:50.865845 31086 server_base.cc:1061] running on GCE node
I20260320 02:58:50.866425 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:50.867664 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:50.868955 31086 hybrid_clock.cc:648] HybridClock initialized: now 1773975530868945 us; error 30 us; skew 500 ppm
Mar 20 02:58:50 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:50.872813 31086 init.cc:377] Logged in from keytab as oryx/127.23.132.66@KRBTEST.COM (short username oryx)
I20260320 02:58:50.874150 31086 webserver.cc:492] Webserver started at http://127.23.132.66:33559/ using document root <none> and password file <none>
I20260320 02:58:50.874688 31086 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:50.874739 31086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:50.875270 31086 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:50.877327 31086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "35f74addbf6c9ca41f053415cf23a457"
server_key_iv: "51aede2130074e66ab8b1d602881f3f5"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.878327 31086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "35f74addbf6c9ca41f053415cf23a457"
server_key_iv: "51aede2130074e66ab8b1d602881f3f5"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.882961 31086 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:58:50.886050 31101 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.887311 31086 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260320 02:58:50.887434 31086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "35f74addbf6c9ca41f053415cf23a457"
server_key_iv: "51aede2130074e66ab8b1d602881f3f5"
server_key_version: "encryptionkey@0"
I20260320 02:58:50.887543 31086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:50.900713 31086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:50.904487 31086 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:50.904696 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:50.905280 31086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:50.906832 31086 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:50.906915 31086 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.906982 31086 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:50.907028 31086 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:50.920473 31086 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:46461
I20260320 02:58:50.920488 31214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:46461 every 8 connection(s)
I20260320 02:58:50.921897 31086 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:50.922602 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 31086
I20260320 02:58:50.922726 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:58:50.923018 24081 external_mini_cluster.cc:1468] Setting key 1fdd60f79546b68e352f1e3fe5098e7d
WARNING: no policy specified for oryx/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.67@KRBTEST.COM" created.
Mar 20 02:58:50 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:58:50.945868 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:50.924504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:35645 (local address 127.23.132.126:42639)
0320 02:58:50.924732 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:50.924737 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:50.925831 (+ 1094us) server_negotiation.cc:408] Connection header received
0320 02:58:50.926814 (+ 983us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:50.926818 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:50.926987 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:50.927183 (+ 196us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:50.929048 (+ 1865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.929554 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:50.930643 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.931194 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:50.940644 (+ 9450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:50.940689 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:50.940693 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:50.940751 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:50.943002 (+ 2251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:50.943626 (+ 624us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:50.943630 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:50.943632 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:50.943692 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:50.944116 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:50.944119 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:50.944121 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:50.944346 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:50.944455 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:50.945489 (+ 1034us) server_negotiation.cc:300] Negotiation successful
0320 02:58:50.945642 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260320 02:58:50.946679 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:50.924830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:35645)
0320 02:58:50.925602 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:50.925692 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:50.926567 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:50.927376 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:50.927385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:50.928007 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:50.928824 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:50.928842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.929754 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:50.929759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:50.930380 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:50.930391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:50.932838 (+ 2447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:50.934355 (+ 1517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:50.934392 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:50.940305 (+ 5913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:50.943196 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:50.943203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:50.943215 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:50.943494 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:50.943837 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:50.943841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:50.943843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:50.944001 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:50.944560 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:50.944569 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:50.944972 (+ 403us) client_negotiation.cc:769] Sending connection context
0320 02:58:50.945446 (+ 474us) client_negotiation.cc:240] Negotiation successful
0320 02:58:50.945896 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":199,"threads_started":1}
I20260320 02:58:50.948333 31215 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42639
I20260320 02:58:50.948619 31215 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:50.949291 31215 heartbeater.cc:507] Master 127.23.132.126:42639 requested a full tablet report, sending...
I20260320 02:58:50.950927 30886 ts_manager.cc:194] Registered new tserver with Master: 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461)
I20260320 02:58:50.951699 30886 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:35645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:58:50.994434 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:42639
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:51.109951 31222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:51.110191 31222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:51.110280 31222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:51.113770 31222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:51.113898 31222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:51.114003 31222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:51.119818 31222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:51.121156 31222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:51.122073 31222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:51.128975 31228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.128979 31230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.129014 31227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:51.129532 31222 server_base.cc:1061] running on GCE node
I20260320 02:58:51.129994 31222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:51.130558 31222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:51.131917 31222 hybrid_clock.cc:648] HybridClock initialized: now 1773975531131858 us; error 86 us; skew 500 ppm
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:51.138311 31222 init.cc:377] Logged in from keytab as oryx/127.23.132.67@KRBTEST.COM (short username oryx)
I20260320 02:58:51.139865 31222 webserver.cc:492] Webserver started at http://127.23.132.67:45435/ using document root <none> and password file <none>
I20260320 02:58:51.140477 31222 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:51.140532 31222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:51.140720 31222 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:51.142501 31222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "ac75f983b52d436b89f01d48d87119bd"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "2e58339da37c9199a25d84b72e2df9ed"
server_key_iv: "aa3c12943a354afb6f2a34025e4d1710"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.142984 31222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "ac75f983b52d436b89f01d48d87119bd"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "2e58339da37c9199a25d84b72e2df9ed"
server_key_iv: "aa3c12943a354afb6f2a34025e4d1710"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.147261 31222 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:58:51.149729 31237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.150862 31222 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260320 02:58:51.150964 31222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "ac75f983b52d436b89f01d48d87119bd"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "2e58339da37c9199a25d84b72e2df9ed"
server_key_iv: "aa3c12943a354afb6f2a34025e4d1710"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.151054 31222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:51.184875 31222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:51.188319 31222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:51.188488 31222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:51.189010 31222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:51.189901 31222 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:51.189950 31222 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.189990 31222 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:51.190004 31222 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.203770 31222 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:36043
I20260320 02:58:51.203891 31350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:36043 every 8 connection(s)
I20260320 02:58:51.205667 31222 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:51.213188 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 31222
I20260320 02:58:51.213351 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:58:51.213670 24081 external_mini_cluster.cc:1468] Setting key 047219b78956bbb38877ae9d0407d3c7
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:58:51.220718 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.207837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:60465 (local address 127.23.132.126:42639)
0320 02:58:51.208075 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.208079 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.208859 (+ 780us) server_negotiation.cc:408] Connection header received
0320 02:58:51.209707 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.209711 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.209769 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.210002 (+ 233us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.211280 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.211776 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.212381 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.212547 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.216364 (+ 3817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.216388 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.216390 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.216418 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.217845 (+ 1427us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.218546 (+ 701us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.218551 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.218553 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.218635 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.219031 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.219034 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.219036 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.219270 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.219375 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.220399 (+ 1024us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.220527 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260320 02:58:51.221336 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.207979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:60465)
0320 02:58:51.208700 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.208743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.209509 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.210157 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.210166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.210574 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.211129 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.211141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.211892 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.211896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.212271 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.212277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.212470 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.213606 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.213630 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.215921 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.218091 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.218097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.218107 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.218363 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.218786 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.218789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.218791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.218886 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.219446 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.219454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.219936 (+ 482us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.220278 (+ 342us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.220538 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":230,"threads_started":1}
I20260320 02:58:51.222781 31351 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42639
I20260320 02:58:51.223053 31351 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:51.223754 31351 heartbeater.cc:507] Master 127.23.132.126:42639 requested a full tablet report, sending...
I20260320 02:58:51.225241 30886 ts_manager.cc:194] Registered new tserver with Master: ac75f983b52d436b89f01d48d87119bd (127.23.132.67:36043)
I20260320 02:58:51.225983 30886 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:60465
I20260320 02:58:51.229583 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.132.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:58:51.304431 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.125:34787
--webserver_interface=127.23.132.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:42639,127.23.132.125:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:51.421391 31358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:51.421646 31358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:51.421700 31358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:51.425238 31358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:51.425318 31358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:51.425341 31358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:51.425359 31358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:51.425375 31358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:51.429944 31358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:42639,127.23.132.125:34787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.125:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.132.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
W20260320 02:58:51.430147 31358 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.132.126:42639,127.23.132.125:34787'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260320 02:58:51.431003 31358 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:51.431789 31358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:51.438270 31363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.438305 31366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.438294 31364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:51.439126 31358 server_base.cc:1061] running on GCE node
I20260320 02:58:51.439673 31358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:51.440572 31358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:51.441869 31358 hybrid_clock.cc:648] HybridClock initialized: now 1773975531441786 us; error 78 us; skew 500 ppm
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:51.445155 31358 init.cc:377] Logged in from keytab as oryx/127.23.132.125@KRBTEST.COM (short username oryx)
I20260320 02:58:51.446316 31358 webserver.cc:492] Webserver started at http://127.23.132.125:37959/ using document root <none> and password file <none>
I20260320 02:58:51.446867 31358 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:51.446915 31358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:51.447098 31358 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:51.448734 31358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/instance:
uuid: "33519057c23f4e96b2e9456707a75b3d"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "d23e18c4f006581709908d562794930a"
server_key_iv: "9e7a935b121aaa1ce5466dd7d23c1372"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.449174 31358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal/instance:
uuid: "33519057c23f4e96b2e9456707a75b3d"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "d23e18c4f006581709908d562794930a"
server_key_iv: "9e7a935b121aaa1ce5466dd7d23c1372"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.452728 31358 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260320 02:58:51.455376 31373 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.456620 31358 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 02:58:51.456748 31358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
uuid: "33519057c23f4e96b2e9456707a75b3d"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "d23e18c4f006581709908d562794930a"
server_key_iv: "9e7a935b121aaa1ce5466dd7d23c1372"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.456854 31358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:51.473843 31358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:51.479012 31358 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:51.479219 31358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:51.491325 31358 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.125:34787
I20260320 02:58:51.491341 31425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.125:34787 every 8 connection(s)
I20260320 02:58:51.493074 31358 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/info.pb
I20260320 02:58:51.497110 31426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:51.501083 31426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } has no permanent_uuid. Determining permanent_uuid...
I20260320 02:58:51.502692 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 31358
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.125@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
I20260320 02:58:51.518707 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.503353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.23.132.126:42639)
0320 02:58:51.503501 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.503504 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.504604 (+ 1100us) server_negotiation.cc:408] Connection header received
0320 02:58:51.505521 (+ 917us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.505525 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.505580 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.505669 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.507115 (+ 1446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.507946 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.508917 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.509201 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.513201 (+ 4000us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.513232 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.513235 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.513274 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.515765 (+ 2491us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.516526 (+ 761us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.516529 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.516531 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.516585 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.516906 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.516910 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.516912 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.517115 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.517205 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.518397 (+ 1192us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.518534 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260320 02:58:51.519423 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.503756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56902)
0320 02:58:51.504390 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.504428 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.505325 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.505970 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.505976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.506520 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.506961 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.506977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.508113 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.508117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.508662 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.508673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.508939 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.509571 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.509596 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.512949 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.516008 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.516020 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.516031 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.516384 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.516687 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.516690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.516692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.516815 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.517225 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.517232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.517691 (+ 459us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.518341 (+ 650us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.518630 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":273,"threads_started":1}
I20260320 02:58:51.520076 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.504954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.23.132.125:34787)
0320 02:58:51.505297 (+ 343us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.505301 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.505328 (+ 27us) server_negotiation.cc:408] Connection header received
0320 02:58:51.505445 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.505464 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.505655 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.505837 (+ 182us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.507069 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.508179 (+ 1110us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.509039 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.509260 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.513761 (+ 4501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.513895 (+ 134us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.513912 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.513940 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.516644 (+ 2704us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.517279 (+ 635us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.517289 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.517291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.517380 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.517738 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.517742 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.517744 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.518257 (+ 513us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.518559 (+ 302us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.519401 (+ 842us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.519840 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"spinlock_wait_cycles":1792,"thread_start_us":118,"threads_started":1}
I20260320 02:58:51.521903 31426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.125@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
I20260320 02:58:51.533205 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.522743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60568)
0320 02:58:51.522869 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.523009 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.523085 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.524293 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.524296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.524461 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.524652 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.524658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.525574 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.525577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.526234 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.526244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.526354 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.527089 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.527102 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.528597 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.531225 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.531229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.531230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.531397 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.531947 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.531950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.531952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.532015 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.532514 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.532517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.532600 (+ 83us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.532709 (+ 109us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.532977 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260320 02:58:51.533214 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.523218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.23.132.125:34787)
0320 02:58:51.523385 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.523390 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.523402 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:51.523432 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.523434 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.523475 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.524243 (+ 768us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.524963 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.525447 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.526460 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.526895 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.528788 (+ 1893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.528807 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.528810 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.528831 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.530965 (+ 2134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.531577 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.531582 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.531583 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.531666 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.532199 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.532203 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.532204 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.532376 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.532466 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.532862 (+ 396us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.533064 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":86}
I20260320 02:58:51.536872 31426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Bootstrap starting.
I20260320 02:58:51.539595 31426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:51.540333 31426 log.cc:826] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:51.543314 31426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: No bootstrap required, opened a new log
I20260320 02:58:51.547641 31426 raft_consensus.cc:359] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } }
I20260320 02:58:51.548117 31426 raft_consensus.cc:385] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:51.548276 31426 raft_consensus.cc:740] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33519057c23f4e96b2e9456707a75b3d, State: Initialized, Role: FOLLOWER
I20260320 02:58:51.549006 31426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } }
I20260320 02:58:51.549970 31432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } } }
I20260320 02:58:51.550455 31432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:51.551236 31426 sys_catalog.cc:565] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: configured and running, proceeding with master startup.
W20260320 02:58:51.558614 31443 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:58:51.558779 31443 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:58:51.563722 31358 master_runner.cc:428] Detected that this master 33519057c23f4e96b2e9456707a75b3d is joining an existing cluster
I20260320 02:58:51.563800 31358 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.132.125:34787
I20260320 02:58:51.574064 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.566051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.23.132.126:42639)
0320 02:58:51.566182 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.566187 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.566451 (+ 264us) server_negotiation.cc:408] Connection header received
0320 02:58:51.566571 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.566574 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.566630 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.566704 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.567592 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.568564 (+ 972us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.569336 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.569514 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.570290 (+ 776us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.570414 (+ 124us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.570421 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.570446 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.572318 (+ 1872us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.572834 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.572838 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.572839 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.572885 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.573139 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.573143 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.573144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.573341 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.573457 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.573689 (+ 232us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.573855 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260320 02:58:51.574657 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.565986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56912)
0320 02:58:51.566366 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.566377 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.566476 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.566734 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.566737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.566951 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.567271 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.567278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.568699 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.568701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.569208 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.569218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.569342 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.569871 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.569887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.570176 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.572519 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.572524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.572527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.572728 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.572969 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.572972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.572973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.573020 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.573431 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.573434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.573534 (+ 100us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.573628 (+ 94us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.573854 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.23.132.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.124@KRBTEST.COM" created.
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.126@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
I20260320 02:58:51.591177 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.566529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60584)
0320 02:58:51.566866 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.566891 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.567015 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.567243 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.567246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.567430 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.567715 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.567723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.568669 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.568673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.569245 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.569254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.569672 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.586430 (+ 16758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.586449 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.586881 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.589781 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.589809 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.589812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.590026 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.590253 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.590256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.590258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.590301 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.590653 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.590655 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.590723 (+ 68us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.590816 (+ 93us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.590966 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260320 02:58:51.591965 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.566060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.23.132.125:34787)
0320 02:58:51.566273 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.566277 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.566997 (+ 720us) server_negotiation.cc:408] Connection header received
0320 02:58:51.567089 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.567092 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.567130 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.567203 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.567870 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.568534 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.569362 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.569571 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.587102 (+ 17531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.587123 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.587127 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.587150 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.589645 (+ 2495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.590121 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.590124 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.590126 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.590178 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.590368 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.590371 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.590373 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.590544 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.590620 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.591649 (+ 1029us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.591821 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260320 02:58:51.594681 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.581268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.23.132.125:34787)
0320 02:58:51.583554 (+ 2286us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.583558 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.583574 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:58:51.583660 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.583664 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.583726 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.583823 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.584941 (+ 1118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.585758 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.586595 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.587220 (+ 625us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.591044 (+ 3824us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.591066 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.591069 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.591094 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.592884 (+ 1790us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.593405 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.593408 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.593409 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.593445 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.593777 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.593780 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.593781 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.593981 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.594076 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.594404 (+ 328us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.594514 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":111,"threads_started":1}
I20260320 02:58:51.594684 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.581143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60596)
0320 02:58:51.581583 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.581623 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.581814 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.583908 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.583915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.584457 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.584790 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.584801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.585909 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.585914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.586467 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.586478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.586624 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.587402 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.587418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.590898 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.593040 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.593052 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.593057 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.593305 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.593536 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.593540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.593543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.593610 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.594098 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.594105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.594218 (+ 113us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.594345 (+ 127us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.594521 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260320 02:58:51.596629 30886 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.23.132.125:34787: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "33519057c23f4e96b2e9456707a75b3d"
member_type: NON_VOTER
last_known_addr {
host: "127.23.132.125"
port: 34787
}
attrs {
promote: true
}
}
dest_uuid: "a391b242632f4333b94724207e5d6889"
cas_config_opid_index: -1
I20260320 02:58:51.597538 30886 consensus_queue.cc:237] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } }
I20260320 02:58:51.600575 31399 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:51.602294 31399 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Refusing update from remote peer a391b242632f4333b94724207e5d6889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260320 02:58:51.602175 31456 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 33519057c23f4e96b2e9456707a75b3d (127.23.132.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } }
I20260320 02:58:51.603173 31457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:51.604507 31456 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.23.132.125:34787
I20260320 02:58:51.605090 31358 master.cc:561] Master@127.23.132.125:34787 shutting down...
I20260320 02:58:51.605058 31458 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } }
I20260320 02:58:51.605247 31458 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
W20260320 02:58:51.608875 30873 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260320 02:58:51.610924 30873 consensus_peers.cc:597] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 -> Peer 33519057c23f4e96b2e9456707a75b3d (127.23.132.125:34787): Couldn't send request to peer 33519057c23f4e96b2e9456707a75b3d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260320 02:58:51.613294 31358 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:51.613781 31358 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:51.614109 31358 tablet_replica.cc:333] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.124@KRBTEST.COM" created.
I20260320 02:58:51.633519 31358 master.cc:583] Master@127.23.132.125:34787 shutdown complete.
I20260320 02:58:51.633577 31358 master_runner.cc:305] Clearing existing system tablet
I20260320 02:58:51.633781 31358 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:51.634024 31358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:51.636734 31358 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260320 02:58:51.639077 31465 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.639962 31358 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:51.640122 31358 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 02:58:51.640172 31358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
uuid: "33519057c23f4e96b2e9456707a75b3d"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "d23e18c4f006581709908d562794930a"
server_key_iv: "9e7a935b121aaa1ce5466dd7d23c1372"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.642000 31358 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:51.644315 31358 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260320 02:58:51.644397 31358 log.cc:1199] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260320 02:58:51.644600 31358 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Deleting consensus metadata
I20260320 02:58:51.644807 31358 master_runner.cc:315] Copying system tablet from 127.23.132.126:42639
I20260320 02:58:51.646086 31358 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: tablet copy: Beginning tablet copy session from remote peer at address 127.23.132.126:42639
I20260320 02:58:51.653165 31079 heartbeater.cc:499] Master 127.23.132.126:42639 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:58:51.666531 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.124:42609
--webserver_interface=127.23.132.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260320 02:58:51.668082 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.646804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56918)
0320 02:58:51.647473 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.647490 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.647675 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.648289 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.648294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.648519 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.648788 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.648796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.650163 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.650167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.650581 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.650591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.651066 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.651921 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.651935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.652273 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.666394 (+ 14121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.666400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.666402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.666677 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.667003 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.667006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.667008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.667063 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.667492 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.667495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.667597 (+ 102us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.667702 (+ 105us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.667889 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":471,"threads_started":1}
I20260320 02:58:51.668370 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.646966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.23.132.126:42639)
0320 02:58:51.647475 (+ 509us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.647480 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.647575 (+ 95us) server_negotiation.cc:408] Connection header received
0320 02:58:51.647956 (+ 381us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.647961 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.648036 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.648264 (+ 228us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.648988 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.649708 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.651463 (+ 1755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.651685 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.652390 (+ 705us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.652412 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.652416 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.652437 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.666167 (+ 13730us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.666832 (+ 665us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.666836 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.666838 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.666897 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.667163 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.667167 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.667170 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.667389 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.667500 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.667809 (+ 309us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.667975 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260320 02:58:51.668936 30906 tablet_copy_service.cc:140] P a391b242632f4333b94724207e5d6889: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 33519057c23f4e96b2e9456707a75b3d ({username='oryx', principal='oryx/127.23.132.125@KRBTEST.COM'} at 127.0.0.1:56918)
I20260320 02:58:51.669126 30906 tablet_copy_service.cc:161] P a391b242632f4333b94724207e5d6889: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 33519057c23f4e96b2e9456707a75b3d at {username='oryx', principal='oryx/127.23.132.125@KRBTEST.COM'} at 127.0.0.1:56918: session id = 33519057c23f4e96b2e9456707a75b3d-00000000000000000000000000000000
I20260320 02:58:51.670450 30906 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Tablet Copy: opened 0 blocks and 1 log segments
I20260320 02:58:51.671927 31358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260320 02:58:51.673736 31358 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: tablet copy: Starting download of 0 data blocks...
I20260320 02:58:51.673921 31358 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: tablet copy: Starting download of 1 WAL segments...
I20260320 02:58:51.675531 31358 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260320 02:58:51.677466 30906 tablet_copy_service.cc:342] P a391b242632f4333b94724207e5d6889: Request end of tablet copy session 33519057c23f4e96b2e9456707a75b3d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.132.125@KRBTEST.COM'} at 127.0.0.1:56918
I20260320 02:58:51.677605 30906 tablet_copy_service.cc:434] P a391b242632f4333b94724207e5d6889: ending tablet copy session 33519057c23f4e96b2e9456707a75b3d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 33519057c23f4e96b2e9456707a75b3d
W20260320 02:58:51.684722 31358 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260320 02:58:51.687023 31358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:51.689226 31474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.689275 31476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.689476 31473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:51.689747 31358 server_base.cc:1061] running on GCE node
I20260320 02:58:51.690168 31358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:51.690462 31358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:51.691720 31358 hybrid_clock.cc:648] HybridClock initialized: now 1773975531691680 us; error 55 us; skew 500 ppm
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:51.693166 31358 init.cc:377] Logged in from keytab as oryx/127.23.132.125@KRBTEST.COM (short username oryx)
I20260320 02:58:51.693627 31358 webserver.cc:492] Webserver started at http://127.23.132.125:45617/ using document root <none> and password file <none>
I20260320 02:58:51.693837 31358 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:51.693953 31358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:51.696153 31358 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260320 02:58:51.697912 31483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.698737 31358 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260320 02:58:51.698839 31358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
uuid: "33519057c23f4e96b2e9456707a75b3d"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "d23e18c4f006581709908d562794930a"
server_key_iv: "9e7a935b121aaa1ce5466dd7d23c1372"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.699121 31358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:51.705227 31358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:51.707875 31358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:51.714960 31358 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.125:34787
I20260320 02:58:51.715618 31358 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-1/data/info.pb
I20260320 02:58:51.717182 31536 sys_catalog.cc:263] Verifying existing consensus state
I20260320 02:58:51.717944 31535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.125:34787 every 8 connection(s)
I20260320 02:58:51.718075 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Bootstrap starting.
I20260320 02:58:51.728067 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260320 02:58:51.728624 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: Bootstrap complete.
I20260320 02:58:51.729372 31536 raft_consensus.cc:359] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } }
I20260320 02:58:51.729487 31536 raft_consensus.cc:740] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33519057c23f4e96b2e9456707a75b3d, State: Initialized, Role: LEARNER
I20260320 02:58:51.729596 31536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } }
I20260320 02:58:51.730540 31536 sys_catalog.cc:565] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:51.730444 31539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } }
I20260320 02:58:51.730813 31539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: LEARNER
I20260320 02:58:51.734370 31550 catalog_manager.cc:1269] Loaded cluster ID: 0460188e15544f349f52287fff0d88cd
I20260320 02:58:51.734503 31550 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: loading cluster ID for follower catalog manager: success
I20260320 02:58:51.736687 31550 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: acquiring CA information for follower catalog manager: success
I20260320 02:58:51.738370 31550 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260320 02:58:51.833755 31472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:51.834049 31472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:51.834107 31472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:51.837769 31472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:58:51.837869 31472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:51.837893 31472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:51.837918 31472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:58:51.837939 31472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:58:51.842969 31472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.124:42609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.132.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:51.844129 31472 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:51.845165 31472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:51.852833 31556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.852833 31555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:51.853097 31558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:51.854041 31472 server_base.cc:1061] running on GCE node
I20260320 02:58:51.854621 31472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:51.855746 31472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:51.857111 31472 hybrid_clock.cc:648] HybridClock initialized: now 1773975531857051 us; error 92 us; skew 500 ppm
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:51.861050 31472 init.cc:377] Logged in from keytab as oryx/127.23.132.124@KRBTEST.COM (short username oryx)
I20260320 02:58:51.862564 31472 webserver.cc:492] Webserver started at http://127.23.132.124:42731/ using document root <none> and password file <none>
I20260320 02:58:51.863134 31472 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:51.863189 31472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:51.863370 31472 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:58:51.865039 31472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/instance:
uuid: "8221429cd61248029bfa740ab799e3f6"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "9b753b7bf63906601f6efe7cd93d8173"
server_key_iv: "99742d9242ef085ec3ca276a26a7b451"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.865602 31472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal/instance:
uuid: "8221429cd61248029bfa740ab799e3f6"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "9b753b7bf63906601f6efe7cd93d8173"
server_key_iv: "99742d9242ef085ec3ca276a26a7b451"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.869163 31472 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260320 02:58:51.871806 31565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:51.873371 31472 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:58:51.873616 31472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
uuid: "8221429cd61248029bfa740ab799e3f6"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "9b753b7bf63906601f6efe7cd93d8173"
server_key_iv: "99742d9242ef085ec3ca276a26a7b451"
server_key_version: "encryptionkey@0"
I20260320 02:58:51.873876 31472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:51.888995 31472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:51.892504 31472 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:51.892699 31472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:51.903147 31617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.124:42609 every 8 connection(s)
I20260320 02:58:51.903147 31472 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.124:42609
I20260320 02:58:51.904379 31472 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/info.pb
I20260320 02:58:51.906546 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 31472
I20260320 02:58:51.907982 31618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:51.912720 31618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } has no permanent_uuid. Determining permanent_uuid...
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.132.124@KRBTEST.COM
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.124@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:58:51.923659 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.908365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.23.132.124:42609)
0320 02:58:51.908913 (+ 548us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.908927 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.908977 (+ 50us) server_negotiation.cc:408] Connection header received
0320 02:58:51.910296 (+ 1319us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.910320 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.910615 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.910995 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.912381 (+ 1386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.913259 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.914331 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.914906 (+ 575us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.917406 (+ 2500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.917460 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.917470 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.917512 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.919798 (+ 2286us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.920283 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.920289 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.920295 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.920375 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.920621 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.920624 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.920627 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.921051 (+ 424us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.921233 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.922566 (+ 1333us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.922847 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":145,"threads_started":1}
I20260320 02:58:51.924750 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.914391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.23.132.126:42639)
0320 02:58:51.914539 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.914543 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.915183 (+ 640us) server_negotiation.cc:408] Connection header received
0320 02:58:51.915238 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.915241 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.915299 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.915391 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.916549 (+ 1158us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.917263 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.918196 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.918338 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.920544 (+ 2206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.920564 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.920567 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.920592 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.922389 (+ 1797us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.923229 (+ 840us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.923234 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.923235 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.923305 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.923705 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.923708 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.923709 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.923912 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.924012 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.924496 (+ 484us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.924600 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260320 02:58:51.924808 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.914154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56920)
0320 02:58:51.915009 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.915048 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.915157 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.915693 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.915697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.916072 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.916399 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.916409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.917401 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.917405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.918041 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.918053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.918185 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.918712 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.918730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.920424 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.922530 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.922537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.922540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.923055 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.923404 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.923408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.923410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.923463 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.924070 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.924075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.924273 (+ 198us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.924412 (+ 139us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.924586 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":136,"threads_started":1}
I20260320 02:58:51.927237 31618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.124@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
I20260320 02:58:51.939872 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.928082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60610)
0320 02:58:51.928222 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.928235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.928306 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.928833 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.928837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.929065 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.929315 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.929323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.930564 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.930618 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.931478 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.931490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.931620 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.932583 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.932609 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.935186 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.938130 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.938134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.938136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.938329 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.938767 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.938770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.938772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.938835 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.939379 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.939381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.939464 (+ 83us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.939544 (+ 80us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.939668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260320 02:58:51.939924 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.928230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.23.132.125:34787)
0320 02:58:51.928472 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.928475 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.928487 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:51.928523 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.928526 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.928565 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.928782 (+ 217us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.929511 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.930408 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.931682 (+ 1274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.931881 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.935469 (+ 3588us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.935581 (+ 112us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.935593 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.935629 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.937714 (+ 2085us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.938500 (+ 786us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.938504 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.938506 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.938574 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.939037 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.939040 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.939041 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.939225 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.939309 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.939653 (+ 344us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.939792 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260320 02:58:51.941269 31618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } has no permanent_uuid. Determining permanent_uuid...
Mar 20 02:58:51 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975531, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.124@KRBTEST.COM for oryx/127.23.132.124@KRBTEST.COM
I20260320 02:58:51.952186 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.942351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.23.132.124:42609)
0320 02:58:51.942454 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.942457 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.942468 (+ 11us) server_negotiation.cc:408] Connection header received
0320 02:58:51.942495 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.942497 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.942531 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.942596 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:51.943178 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.944002 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.944711 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.944868 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.948063 (+ 3195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:51.948087 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:51.948090 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:51.948119 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:51.949968 (+ 1849us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.950431 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.950434 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.950435 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.950479 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:51.950826 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:51.950829 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:51.950830 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:51.951003 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:51.951087 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.951815 (+ 728us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.952058 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260320 02:58:51.952375 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.942077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44376)
0320 02:58:51.942208 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.942223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.942302 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.942644 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.942648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.942816 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:51.943018 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.943025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.944143 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.944146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.944503 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.944509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.944790 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.945839 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:51.945856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:51.947797 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:51.950172 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.950175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.950177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.950321 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.950575 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:51.950577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:51.950579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:51.950625 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:51.951217 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:51.951221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:51.951448 (+ 227us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.951692 (+ 244us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.952196 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":92}
I20260320 02:58:51.955709 31618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Bootstrap starting.
I20260320 02:58:51.955989 31215 heartbeater.cc:499] Master 127.23.132.126:42639 was elected leader, sending a full tablet report...
I20260320 02:58:51.959271 31618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:51.960052 31618 log.cc:826] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:51.962034 31618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: No bootstrap required, opened a new log
I20260320 02:58:51.964716 31618 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } }
I20260320 02:58:51.964883 31618 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:51.964927 31618 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221429cd61248029bfa740ab799e3f6, State: Initialized, Role: FOLLOWER
I20260320 02:58:51.965287 31618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } }
I20260320 02:58:51.965737 31624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } } }
I20260320 02:58:51.965886 31624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:51.966431 31618 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: configured and running, proceeding with master startup.
W20260320 02:58:51.970920 31635 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:58:51.971009 31635 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:58:51.976486 31472 master_runner.cc:428] Detected that this master 8221429cd61248029bfa740ab799e3f6 is joining an existing cluster
I20260320 02:58:51.976555 31472 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.132.124:42609
I20260320 02:58:51.982800 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.979454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56936)
0320 02:58:51.979797 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.979815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.979973 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.980371 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.980375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.980389 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:51.980592 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.980601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.981590 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.981593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.982216 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.982224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.982324 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.982392 (+ 68us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.982445 (+ 53us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.982512 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260320 02:58:51.983151 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.980081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.23.132.126:42639)
0320 02:58:51.980186 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.980189 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.980198 (+ 9us) server_negotiation.cc:408] Connection header received
0320 02:58:51.980238 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.980241 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.980275 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.980336 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:51.980820 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.981489 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.982327 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.982890 (+ 563us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.982957 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.983008 (+ 51us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.983064 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260320 02:58:51.988067 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.984277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60636)
0320 02:58:51.984434 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.984447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.984514 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.984877 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.984881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:51.984899 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:51.985106 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.985113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.986416 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.986422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:51.987450 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:51.987467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.987767 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.987789 (+ 22us) client_negotiation.cc:769] Sending connection context
0320 02:58:51.987843 (+ 54us) client_negotiation.cc:240] Negotiation successful
0320 02:58:51.987937 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260320 02:58:51.988663 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.984401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.23.132.125:34787)
0320 02:58:51.984638 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.984642 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.984653 (+ 11us) server_negotiation.cc:408] Connection header received
0320 02:58:51.984692 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.984695 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.984740 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.984839 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:51.985337 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.986235 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:51.987761 (+ 1526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:51.988320 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:51.988458 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:51.988506 (+ 48us) server_negotiation.cc:300] Negotiation successful
0320 02:58:51.988563 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260320 02:58:52.021286 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.980494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.23.132.124:42609)
0320 02:58:51.980771 (+ 277us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.980774 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.980943 (+ 169us) server_negotiation.cc:408] Connection header received
0320 02:58:51.981059 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.981062 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.981091 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.981129 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:52.010799 (+ 29670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.011476 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.012256 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.012429 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.014816 (+ 2387us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:52.014834 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:52.014837 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:52.014859 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:52.017937 (+ 3078us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.019620 (+ 1683us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.019625 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.019627 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.019697 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.020192 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.020196 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.020198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.020372 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:52.020434 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:52.020738 (+ 304us) server_negotiation.cc:300] Negotiation successful
0320 02:58:52.021127 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"spinlock_wait_cycles":435072}
I20260320 02:58:52.021956 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.980588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.23.132.126:42639)
0320 02:58:51.981102 (+ 514us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.981105 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.981118 (+ 13us) server_negotiation.cc:408] Connection header received
0320 02:58:51.981149 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.981153 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.981192 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.981276 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:52.010792 (+ 29516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.011654 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.012880 (+ 1226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.013152 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.014519 (+ 1367us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:52.014540 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:52.014543 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:52.014570 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:52.017060 (+ 2490us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.019053 (+ 1993us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.019071 (+ 18us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.019074 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.019133 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.019387 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.019390 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.019392 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.019584 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:52.019691 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:52.021499 (+ 1808us) server_negotiation.cc:300] Negotiation successful
0320 02:58:52.021737 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":94,"threads_started":1}
I20260320 02:58:52.022377 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.979331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56928)
0320 02:58:51.979962 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.979975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.980061 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.981380 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.981383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:52.010250 (+ 28867us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:52.010613 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.010625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.011792 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.011796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:52.012423 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.012432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.012554 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.013321 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:52.013338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:52.014315 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:52.017194 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.017198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.017200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.018753 (+ 1553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.019233 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.019236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.019238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.019295 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.019736 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:52.019739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:52.019830 (+ 91us) client_negotiation.cc:769] Sending connection context
0320 02:58:52.019981 (+ 151us) client_negotiation.cc:240] Negotiation successful
0320 02:58:52.020136 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"spinlock_wait_cycles":14592,"thread_start_us":118,"threads_started":1}
I20260320 02:58:52.022670 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.979741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44384)
0320 02:58:51.980868 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.980881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.980987 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.981227 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.981229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:52.010234 (+ 29005us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:52.010686 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.010694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.011624 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.011627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:52.012137 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.012147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.012552 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.013476 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:52.013492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:52.014681 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:52.018647 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.018651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.018653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.018869 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.019815 (+ 946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.019818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.019820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.020071 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.020510 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:52.020512 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:52.020580 (+ 68us) client_negotiation.cc:769] Sending connection context
0320 02:58:52.021443 (+ 863us) client_negotiation.cc:240] Negotiation successful
0320 02:58:52.022367 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"spinlock_wait_cycles":29184,"thread_start_us":64,"threads_started":1}
I20260320 02:58:52.024960 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.979496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.23.132.125:34787)
0320 02:58:51.980162 (+ 666us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:51.980167 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:51.980398 (+ 231us) server_negotiation.cc:408] Connection header received
0320 02:58:51.980562 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:51.980565 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:51.980614 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:51.980666 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:52.010814 (+ 30148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.011571 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.012651 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.012950 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.014877 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:52.014901 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:52.014905 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:52.014932 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:52.017716 (+ 2784us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.019129 (+ 1413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.019133 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.019135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.019186 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.020391 (+ 1205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.020395 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.020397 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.020600 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:52.020697 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:52.024616 (+ 3919us) server_negotiation.cc:300] Negotiation successful
0320 02:58:52.024799 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577}
I20260320 02:58:52.025369 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:51.979611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60620)
0320 02:58:51.980326 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:51.980339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:51.980459 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:51.980826 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:51.980829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:52.010234 (+ 29405us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:52.010633 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.010643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.011695 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.011698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:52.012527 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.012535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.013100 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.013470 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:52.013487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:52.014752 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:52.018199 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.018203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.018205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.019020 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.020221 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.020224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.020226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.020272 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.021015 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:52.021019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:52.021107 (+ 88us) client_negotiation.cc:769] Sending connection context
0320 02:58:52.021218 (+ 111us) client_negotiation.cc:240] Negotiation successful
0320 02:58:52.021333 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"spinlock_wait_cycles":11264,"thread_start_us":104,"threads_started":1}
Mar 20 02:58:52 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975530, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.126@KRBTEST.COM for oryx/127.23.132.124@KRBTEST.COM
I20260320 02:58:52.041404 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.028535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44390)
0320 02:58:52.028734 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:52.028747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:52.028820 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:52.029190 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:52.029192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:52.029348 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:52.029541 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.029547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.031154 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.031158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:52.031719 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.031729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.031863 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.032404 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:52.032421 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:52.034655 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:52.038981 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.038986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.038989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.039240 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.039519 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.039522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.039654 (+ 132us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.039775 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.040719 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:52.040724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:52.040884 (+ 160us) client_negotiation.cc:769] Sending connection context
0320 02:58:52.040981 (+ 97us) client_negotiation.cc:240] Negotiation successful
0320 02:58:52.041188 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260320 02:58:52.042943 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.028762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.23.132.124:42609)
0320 02:58:52.028881 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:52.028885 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:52.028899 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:58:52.028982 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:52.028985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:52.029036 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:52.029113 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:52.029980 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.030850 (+ 870us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.031989 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.032209 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.034935 (+ 2726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:52.035009 (+ 74us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:52.035017 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:52.035049 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:52.038794 (+ 3745us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.039356 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.039360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.039362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.039418 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.040291 (+ 873us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.040297 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.040299 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.040569 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:52.042068 (+ 1499us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:52.042593 (+ 525us) server_negotiation.cc:300] Negotiation successful
0320 02:58:52.042763 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260320 02:58:52.044252 30886 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.23.132.124:42609: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8221429cd61248029bfa740ab799e3f6"
member_type: NON_VOTER
last_known_addr {
host: "127.23.132.124"
port: 42609
}
attrs {
promote: true
}
}
dest_uuid: "a391b242632f4333b94724207e5d6889"
cas_config_opid_index: 5
I20260320 02:58:52.045289 30886 consensus_queue.cc:237] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } }
I20260320 02:58:52.048126 31592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:52.048048 31456 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8221429cd61248029bfa740ab799e3f6 (127.23.132.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.049849 31456 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.23.132.124:42609
I20260320 02:58:52.050160 31456 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.050350 31456 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:52.050908 31510 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 LEARNER]: Refusing update from remote peer a391b242632f4333b94724207e5d6889: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260320 02:58:52.051291 31592 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Refusing update from remote peer a391b242632f4333b94724207e5d6889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260320 02:58:52.051553 31472 master.cc:561] Master@127.23.132.124:42609 shutting down...
I20260320 02:58:52.051635 31457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260320 02:58:52.052078 31457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260320 02:58:52.055053 30870 consensus_peers.cc:597] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 -> Peer 8221429cd61248029bfa740ab799e3f6 (127.23.132.124:42609): Couldn't send request to peer 8221429cd61248029bfa740ab799e3f6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260320 02:58:52.055202 31510 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8221429cd61248029bfa740ab799e3f6 (127.23.132.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.056809 31647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.057051 31647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: LEARNER
I20260320 02:58:52.057911 31539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a391b242632f4333b94724207e5d6889. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: NON_VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: true } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.058007 31539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: LEARNER
I20260320 02:58:52.059324 31457 raft_consensus.cc:1064] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: attempting to promote NON_VOTER 33519057c23f4e96b2e9456707a75b3d to VOTER
I20260320 02:58:52.059993 31457 consensus_queue.cc:237] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } }
I20260320 02:58:52.062029 31510 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 LEARNER]: Refusing update from remote peer a391b242632f4333b94724207e5d6889: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260320 02:58:52.062714 31456 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260320 02:58:52.062947 30870 consensus_peers.cc:597] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 -> Peer 8221429cd61248029bfa740ab799e3f6 (127.23.132.124:42609): Couldn't send request to peer 8221429cd61248029bfa740ab799e3f6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260320 02:58:52.064280 31472 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:52.064739 31472 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:52.064877 31472 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: stopping tablet replica
I20260320 02:58:52.065448 31458 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33519057c23f4e96b2e9456707a75b3d (127.23.132.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.067112 31458 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.067211 31458 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:52.067517 31457 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.067620 31457 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:52.068622 31510 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33519057c23f4e96b2e9456707a75b3d (127.23.132.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.070716 31539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.070838 31539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:52.080238 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.079788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44392)
0320 02:58:52.080034 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:52.080128 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.124:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260320 02:58:52.080554 30886 master.cc:679] Network error: unable to get registration information for peer 8221429cd61248029bfa740ab799e3f6: Client connection negotiation failed: client connection to 127.23.132.124:42609: connect: Connection refused (error 111)
I20260320 02:58:52.084767 31472 master.cc:583] Master@127.23.132.124:42609 shutdown complete.
I20260320 02:58:52.084851 31472 master_runner.cc:305] Clearing existing system tablet
I20260320 02:58:52.085260 31472 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:52.085357 31472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:52.086774 31472 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260320 02:58:52.087728 31649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:52.088133 31472 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:52.088418 31472 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260320 02:58:52.088508 31472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
uuid: "8221429cd61248029bfa740ab799e3f6"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "9b753b7bf63906601f6efe7cd93d8173"
server_key_iv: "99742d9242ef085ec3ca276a26a7b451"
server_key_version: "encryptionkey@0"
I20260320 02:58:52.090683 31472 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:52.093336 31472 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260320 02:58:52.093439 31472 log.cc:1199] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260320 02:58:52.093704 31472 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Deleting consensus metadata
I20260320 02:58:52.093971 31472 master_runner.cc:315] Copying system tablet from 127.23.132.126:42639
I20260320 02:58:52.095676 31472 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: tablet copy: Beginning tablet copy session from remote peer at address 127.23.132.126:42639
I20260320 02:58:52.105659 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.096651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.23.132.126:42639)
0320 02:58:52.096835 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:52.096840 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:52.096920 (+ 80us) server_negotiation.cc:408] Connection header received
0320 02:58:52.097061 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:52.097065 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:52.097123 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:52.097250 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:52.098075 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.098926 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.099793 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.100008 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.101444 (+ 1436us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:52.101487 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:52.101491 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:52.101529 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:52.103567 (+ 2038us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.104200 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.104205 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.104207 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.104281 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:52.104605 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:52.104608 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:52.104610 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:52.104825 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:52.104957 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:52.105302 (+ 345us) server_negotiation.cc:300] Negotiation successful
0320 02:58:52.105474 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260320 02:58:52.105682 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.096505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56938)
0320 02:58:52.096817 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:52.096832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:52.096962 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:52.097328 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:52.097331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:52.097546 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:52.097841 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.097852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.099149 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.099153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:52.099661 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.099670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.099934 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.100652 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:52.100669 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:52.101237 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:52.103753 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.103757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.103759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.103992 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.104436 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:52.104439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:52.104440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:52.104497 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:52.104977 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:52.104981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:52.105115 (+ 134us) client_negotiation.cc:769] Sending connection context
0320 02:58:52.105225 (+ 110us) client_negotiation.cc:240] Negotiation successful
0320 02:58:52.105387 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260320 02:58:52.106205 30906 tablet_copy_service.cc:140] P a391b242632f4333b94724207e5d6889: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8221429cd61248029bfa740ab799e3f6 ({username='oryx', principal='oryx/127.23.132.124@KRBTEST.COM'} at 127.0.0.1:56938)
I20260320 02:58:52.106307 30906 tablet_copy_service.cc:161] P a391b242632f4333b94724207e5d6889: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8221429cd61248029bfa740ab799e3f6 at {username='oryx', principal='oryx/127.23.132.124@KRBTEST.COM'} at 127.0.0.1:56938: session id = 8221429cd61248029bfa740ab799e3f6-00000000000000000000000000000000
I20260320 02:58:52.107335 30906 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Tablet Copy: opened 0 blocks and 1 log segments
I20260320 02:58:52.109160 31472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260320 02:58:52.111970 31472 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: tablet copy: Starting download of 0 data blocks...
I20260320 02:58:52.112267 31472 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: tablet copy: Starting download of 1 WAL segments...
I20260320 02:58:52.115306 31472 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260320 02:58:52.118587 30906 tablet_copy_service.cc:342] P a391b242632f4333b94724207e5d6889: Request end of tablet copy session 8221429cd61248029bfa740ab799e3f6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.132.124@KRBTEST.COM'} at 127.0.0.1:56938
I20260320 02:58:52.118721 30906 tablet_copy_service.cc:434] P a391b242632f4333b94724207e5d6889: ending tablet copy session 8221429cd61248029bfa740ab799e3f6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8221429cd61248029bfa740ab799e3f6
W20260320 02:58:52.124629 31472 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260320 02:58:52.127801 31472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:52.131247 31656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:52.131307 31659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:52.131464 31657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:52.133097 31472 server_base.cc:1061] running on GCE node
I20260320 02:58:52.133365 31472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:52.134882 31472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:52.136106 31472 hybrid_clock.cc:648] HybridClock initialized: now 1773975532136095 us; error 43 us; skew 500 ppm
Mar 20 02:58:52 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975532, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:52.138885 31472 init.cc:377] Logged in from keytab as oryx/127.23.132.124@KRBTEST.COM (short username oryx)
I20260320 02:58:52.139607 31472 webserver.cc:492] Webserver started at http://127.23.132.124:36245/ using document root <none> and password file <none>
I20260320 02:58:52.139847 31472 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:52.139886 31472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:52.141671 31472 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260320 02:58:52.143383 31666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:52.144120 31472 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260320 02:58:52.144210 31472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
uuid: "8221429cd61248029bfa740ab799e3f6"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "9b753b7bf63906601f6efe7cd93d8173"
server_key_iv: "99742d9242ef085ec3ca276a26a7b451"
server_key_version: "encryptionkey@0"
I20260320 02:58:52.144474 31472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:52.150383 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.149709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44394)
0320 02:58:52.149874 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:52.150286 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.124:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260320 02:58:52.150686 30886 master.cc:679] Network error: unable to get registration information for peer 8221429cd61248029bfa740ab799e3f6: Client connection negotiation failed: client connection to 127.23.132.124:42609: connect: Connection refused (error 111)
I20260320 02:58:52.158337 31472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:52.161334 31472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:52.170013 31472 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.124:42609
I20260320 02:58:52.170074 31718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.124:42609 every 8 connection(s)
I20260320 02:58:52.170840 31472 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/master-2/data/info.pb
I20260320 02:58:52.172703 31719 sys_catalog.cc:263] Verifying existing consensus state
I20260320 02:58:52.173866 31719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Bootstrap starting.
I20260320 02:58:52.187767 31719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260320 02:58:52.188109 31719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Bootstrap complete.
I20260320 02:58:52.188966 31719 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } }
I20260320 02:58:52.189109 31719 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8221429cd61248029bfa740ab799e3f6, State: Initialized, Role: LEARNER
I20260320 02:58:52.189211 31719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } }
I20260320 02:58:52.189715 31719 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:58:52.189716 31722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.189965 31722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: This master's current role is: LEARNER
I20260320 02:58:52.193708 31733 catalog_manager.cc:1269] Loaded cluster ID: 0460188e15544f349f52287fff0d88cd
I20260320 02:58:52.193866 31733 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: loading cluster ID for follower catalog manager: success
I20260320 02:58:52.196787 31733 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: acquiring CA information for follower catalog manager: success
I20260320 02:58:52.198875 31733 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260320 02:58:52.228672 31351 heartbeater.cc:499] Master 127.23.132.126:42639 was elected leader, sending a full tablet report...
I20260320 02:58:52.287806 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.284146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44408)
0320 02:58:52.284286 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:52.284299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:52.284371 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:52.284915 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:52.284918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:52.284938 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:52.285163 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.285168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.286693 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.286697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:52.287442 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:52.287453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.287569 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.287588 (+ 19us) client_negotiation.cc:769] Sending connection context
0320 02:58:52.287623 (+ 35us) client_negotiation.cc:240] Negotiation successful
0320 02:58:52.287679 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260320 02:58:52.288532 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:52.284315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.23.132.124:42609)
0320 02:58:52.284617 (+ 302us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:52.284621 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:52.284632 (+ 11us) server_negotiation.cc:408] Connection header received
0320 02:58:52.284683 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:52.284686 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:52.284769 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:52.284879 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:52.285320 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.286488 (+ 1168us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:52.287677 (+ 1189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:52.288159 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:52.288290 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:52.288338 (+ 48us) server_negotiation.cc:300] Negotiation successful
0320 02:58:52.288401 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":134,"threads_started":1}
I20260320 02:58:52.490958 31693 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260320 02:58:52.491950 31722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a391b242632f4333b94724207e5d6889. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:52.492100 31722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: This master's current role is: LEARNER
I20260320 02:58:53.027472 31735 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: NON_VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: true } } }
I20260320 02:58:53.027825 31735 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:53.029309 31456 raft_consensus.cc:1064] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: attempting to promote NON_VOTER 8221429cd61248029bfa740ab799e3f6 to VOTER
I20260320 02:58:53.029769 31456 consensus_queue.cc:237] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:53.031477 31510 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Refusing update from remote peer a391b242632f4333b94724207e5d6889: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260320 02:58:53.031560 31693 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 LEARNER]: Refusing update from remote peer a391b242632f4333b94724207e5d6889: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260320 02:58:53.032101 31735 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260320 02:58:53.032478 31456 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260320 02:58:53.035818 31456 raft_consensus.cc:2955] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8221429cd61248029bfa740ab799e3f6 (127.23.132.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.036430 31510 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8221429cd61248029bfa740ab799e3f6 (127.23.132.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.037744 31456 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.037921 31456 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:53.038065 31456 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.038146 31456 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:53.038390 31739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.038638 31739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:53.038800 31737 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.038887 31737 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:53.039742 31693 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8221429cd61248029bfa740ab799e3f6 (127.23.132.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.041325 31740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a391b242632f4333b94724207e5d6889" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:53.041432 31740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:53.085155 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.071640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.23.132.125:34787)
0320 02:58:53.072119 (+ 479us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.072124 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.072168 (+ 44us) server_negotiation.cc:408] Connection header received
0320 02:58:53.072353 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.072357 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.072420 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.072540 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.073884 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.074471 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.075339 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.075525 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.080780 (+ 5255us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.080815 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.080819 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.080867 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.083027 (+ 2160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.083496 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.083500 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.083501 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.083564 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.084182 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.084185 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.084187 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.084378 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.084502 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.084832 (+ 330us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.084977 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":77,"threads_started":1}
I20260320 02:58:53.090412 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.086275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56954)
0320 02:58:53.086900 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.086920 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.087047 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.087464 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.087468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.087503 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.087810 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.087817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.089007 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.089010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.089695 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.089703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.090066 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.090125 (+ 59us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.090177 (+ 52us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.090256 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":332,"threads_started":1}
I20260320 02:58:53.091176 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.086438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.23.132.126:42639)
0320 02:58:53.086748 (+ 310us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.086752 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.087026 (+ 274us) server_negotiation.cc:408] Connection header received
0320 02:58:53.087251 (+ 225us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.087254 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.087301 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.087382 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.088039 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.088716 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.090089 (+ 1373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.090453 (+ 364us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.090492 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.090544 (+ 52us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.090689 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260320 02:58:53.097628 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.093359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60648)
0320 02:58:53.093555 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.093614 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.093725 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.094297 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.094300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.094329 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.094518 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.094524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.095769 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.095772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.097197 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.097210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.097352 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.097375 (+ 23us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.097413 (+ 38us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.097496 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260320 02:58:53.098716 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.093484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.23.132.125:34787)
0320 02:58:53.093699 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.093703 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.093719 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:58:53.094038 (+ 319us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.094042 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.094089 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.094182 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.094860 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.095596 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.097500 (+ 1904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.098491 (+ 991us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.098526 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.098570 (+ 44us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.098625 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260320 02:58:53.104103 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.100305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44412)
0320 02:58:53.100561 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.100577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.100660 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.101466 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.101470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.101497 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.101851 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.101864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.102909 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.102914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.103682 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.103693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.103820 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.103843 (+ 23us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.103889 (+ 46us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.103969 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260320 02:58:53.104611 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.100529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.23.132.124:42609)
0320 02:58:53.101136 (+ 607us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.101141 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.101162 (+ 21us) server_negotiation.cc:408] Connection header received
0320 02:58:53.101227 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.101230 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.101294 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.101418 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.102027 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.102723 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.103908 (+ 1185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.104361 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.104400 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.104454 (+ 54us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.104513 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260320 02:58:53.117081 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.106872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.23.132.124:42609)
0320 02:58:53.106984 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.106987 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.107002 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:58:53.107432 (+ 430us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.107436 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.107511 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.107608 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.108526 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.109217 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.110097 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.110308 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.111241 (+ 933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.111259 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.111261 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.111285 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.114819 (+ 3534us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.115581 (+ 762us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.115584 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.115585 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.115635 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.116102 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.116115 (+ 13us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.116116 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.116288 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.116418 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.116704 (+ 286us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.116901 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260320 02:58:53.120946 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.118155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:56970)
0320 02:58:53.118397 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.118409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.118485 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.118802 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.118812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.118830 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.119045 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.119051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.119919 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.119922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.120543 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.120550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.120656 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.120678 (+ 22us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.120726 (+ 48us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.120791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260320 02:58:53.121341 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.118455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.23.132.126:42639)
0320 02:58:53.118608 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.118611 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.118623 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:53.118672 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.118674 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.118713 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.118774 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.119149 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.119768 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.120665 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.121103 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.121144 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.121192 (+ 48us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.121250 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260320 02:58:53.125579 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.123103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.0.0.1:60654)
0320 02:58:53.123264 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.123279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.123372 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.123668 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.123671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.123693 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.123852 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.123857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.124564 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.124568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.125215 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.125223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.125330 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.125350 (+ 20us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.125397 (+ 47us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.125456 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260320 02:58:53.126173 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.123284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.23.132.125:34787)
0320 02:58:53.123474 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.123478 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.123490 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:53.123532 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.123535 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.123576 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.123642 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.123960 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.124456 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.125331 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.125754 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.125823 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.125876 (+ 53us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.125935 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260320 02:58:53.130417 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.127808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.0.0.1:44424)
0320 02:58:53.127920 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.127934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.128006 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.128301 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.128304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.128321 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.128539 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.128548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.129318 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.129322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.130119 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.130128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.130227 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.130243 (+ 16us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.130275 (+ 32us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.130322 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260320 02:58:53.131336 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.127978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.23.132.124:42609)
0320 02:58:53.128102 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.128105 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.128116 (+ 11us) server_negotiation.cc:408] Connection header received
0320 02:58:53.128148 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.128150 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.128184 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.128254 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.128704 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.129185 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.130313 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.130967 (+ 654us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.131028 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.131095 (+ 67us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.131166 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260320 02:58:53.133684 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30950
I20260320 02:58:53.140030 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:40327
--local_ip_for_outbound_sockets=127.23.132.65
--tserver_master_addrs=127.23.132.126:42639
--webserver_port=35289
--webserver_interface=127.23.132.65
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:58:53.274788 31747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:53.275411 31747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:53.275566 31747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:53.281692 31747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:53.281822 31747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:53.281953 31747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:58:53.288215 31747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:40327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=35289
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:53.289391 31747 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:53.290333 31747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:53.299211 31753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:53.299217 31752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:53.300050 31755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:53.300127 31747 server_base.cc:1061] running on GCE node
I20260320 02:58:53.301182 31747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:53.302060 31747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:53.303267 31747 hybrid_clock.cc:648] HybridClock initialized: now 1773975533303152 us; error 139 us; skew 500 ppm
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:53.306520 31747 init.cc:377] Logged in from keytab as oryx/127.23.132.65@KRBTEST.COM (short username oryx)
I20260320 02:58:53.307760 31747 webserver.cc:492] Webserver started at http://127.23.132.65:35289/ using document root <none> and password file <none>
I20260320 02:58:53.308341 31747 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:53.308410 31747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:53.313882 31747 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260320 02:58:53.318677 31762 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.320202 31747 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260320 02:58:53.320348 31747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "3afdd0e581c244e6a0a63ca52c94a592"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "e84e58655969fdf79bad76f353d31494"
server_key_iv: "b16b1d49af832e1cc49501000f92d014"
server_key_version: "encryptionkey@0"
I20260320 02:58:53.320880 31747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:53.361052 31747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:53.364513 31747 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:53.364717 31747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:53.365301 31747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:53.366300 31747 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:53.366353 31747 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.366405 31747 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:53.366432 31747 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.383565 31747 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:40327
I20260320 02:58:53.383646 31875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:40327 every 8 connection(s)
I20260320 02:58:53.384595 31747 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:58:53.389657 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 31747
I20260320 02:58:53.389958 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 31086
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.124@KRBTEST.COM
I20260320 02:58:53.401080 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:46461
--local_ip_for_outbound_sockets=127.23.132.66
--tserver_master_addrs=127.23.132.126:42639
--webserver_port=33559
--webserver_interface=127.23.132.66
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260320 02:58:53.405961 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.386635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:50897 (local address 127.23.132.125:34787)
0320 02:58:53.386851 (+ 216us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.386856 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.387546 (+ 690us) server_negotiation.cc:408] Connection header received
0320 02:58:53.388823 (+ 1277us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.388829 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.388903 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.388988 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.393194 (+ 4206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.394093 (+ 899us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.395203 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.395503 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.400185 (+ 4682us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.400211 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.400215 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.400246 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.402923 (+ 2677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404202 (+ 1279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.404206 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.404208 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.404270 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404575 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.404578 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.404580 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.404789 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.404853 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.405608 (+ 755us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.405766 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260320 02:58:53.405961 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.388131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:42555 (local address 127.23.132.126:42639)
0320 02:58:53.388266 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.388270 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.388696 (+ 426us) server_negotiation.cc:408] Connection header received
0320 02:58:53.388893 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.388897 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.388963 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.389056 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.393522 (+ 4466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.394425 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.395409 (+ 984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.395593 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.400210 (+ 4617us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.400231 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.400234 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.400263 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.402943 (+ 2680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404203 (+ 1260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.404207 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.404208 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.404270 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404575 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.404578 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.404580 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.404779 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.404853 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.405608 (+ 755us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.405766 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260320 02:58:53.405962 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.386816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:50067 (local address 127.23.132.124:42609)
0320 02:58:53.386959 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.386963 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.387542 (+ 579us) server_negotiation.cc:408] Connection header received
0320 02:58:53.389415 (+ 1873us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.389418 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.389475 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.389558 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.396573 (+ 7015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.397524 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.398646 (+ 1122us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.399195 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.401373 (+ 2178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.401395 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.401398 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.401425 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.403516 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404177 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.404181 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.404183 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.404234 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404572 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.404575 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.404577 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.404831 (+ 254us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.404901 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.405617 (+ 716us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.405762 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260320 02:58:53.407441 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.386959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.23.132.65:50067)
0320 02:58:53.387399 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.387442 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.389244 (+ 1802us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x558937316790. stack: 00007fd78704fac1 00007fd7870500b1 00007fd786b2cadd 000055890d0d037e 000055890d0d03d1 000055890d0d4690 00007fd78704692f 00007fd787045b15 00007fd78acdcc83 00007fd78acef111 00007fd78ace3f1c 00007fd787c0b965 00007fd787c09d57 00007fd787c4fda9 00007fd787c50cd8 00007fd787c682e2
0320 02:58:53.389266 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.389679 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.389683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.395348 (+ 5665us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.396084 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.396183 (+ 99us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.397773 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.397778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.398423 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.398435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.398538 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.399305 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.399324 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.401232 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.403657 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.403714 (+ 57us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.403718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.404057 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.404337 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.404341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.404460 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.404972 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.404975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.405283 (+ 308us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.406198 (+ 915us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.406841 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":213,"spinlock_wait_cycles":2333184,"thread_start_us":117,"threads_started":1}
I20260320 02:58:53.407456 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.388105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:42555)
0320 02:58:53.388476 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.388571 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.388766 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.389289 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.389310 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.391197 (+ 1887us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.393286 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.393302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.394598 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.394602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.395270 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.395280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.395904 (+ 624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.397018 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.397069 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.399975 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.403238 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.403245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.403260 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.404060 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.404382 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.404387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.404475 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.404938 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.404945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.405283 (+ 338us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.406198 (+ 915us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.406539 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"spinlock_wait_cycles":143104,"thread_start_us":85,"threads_started":1}
I20260320 02:58:53.407456 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.387022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.23.132.65:50897)
0320 02:58:53.387400 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.387442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.388565 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.389289 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.389310 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.391220 (+ 1910us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.392649 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.392680 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.394401 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.394417 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.395028 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.395040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.395904 (+ 864us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.397557 (+ 1653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.397573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.399975 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.403274 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.403277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.403279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.404060 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.404382 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.404385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.404387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.404475 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.404938 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.404944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.405283 (+ 339us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.406633 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.406925 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":127,"spinlock_wait_cycles":177024,"thread_start_us":138,"threads_started":1}
I20260320 02:58:53.409068 31877 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42639
I20260320 02:58:53.409075 31876 heartbeater.cc:344] Connected to a master server at 127.23.132.124:42609
I20260320 02:58:53.409574 31877 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.409574 31876 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.410472 31876 heartbeater.cc:507] Master 127.23.132.124:42609 requested a full tablet report, sending...
I20260320 02:58:53.410472 31877 heartbeater.cc:507] Master 127.23.132.126:42639 requested a full tablet report, sending...
I20260320 02:58:53.410564 31878 heartbeater.cc:344] Connected to a master server at 127.23.132.125:34787
I20260320 02:58:53.410703 31878 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.410998 31878 heartbeater.cc:507] Master 127.23.132.125:34787 requested a full tablet report, sending...
I20260320 02:58:53.412130 30886 ts_manager.cc:194] Re-registered known tserver with Master: 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327)
I20260320 02:58:53.412403 31683 ts_manager.cc:194] Registered new tserver with Master: 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327)
I20260320 02:58:53.412436 31500 ts_manager.cc:194] Registered new tserver with Master: 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327)
I20260320 02:58:53.412932 30886 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:42555
W20260320 02:58:53.544498 31883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:53.544762 31883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:53.544834 31883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:53.551548 31883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:53.551704 31883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:53.551808 31883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:58:53.560118 31883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:46461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=33559
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:53.561893 31883 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:53.563120 31883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:53.572852 31888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:53.572860 31891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:53.573213 31883 server_base.cc:1061] running on GCE node
W20260320 02:58:53.572856 31889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:53.573746 31883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:53.574802 31883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:53.576012 31883 hybrid_clock.cc:648] HybridClock initialized: now 1773975533575995 us; error 38 us; skew 500 ppm
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:53.579800 31883 init.cc:377] Logged in from keytab as oryx/127.23.132.66@KRBTEST.COM (short username oryx)
I20260320 02:58:53.581121 31883 webserver.cc:492] Webserver started at http://127.23.132.66:33559/ using document root <none> and password file <none>
I20260320 02:58:53.581874 31883 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:53.581959 31883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:53.586469 31883 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:53.589176 31898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.590708 31883 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:53.590839 31883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3"
format_stamp: "Formatted at 2026-03-20 02:58:50 on dist-test-slave-v3hj"
server_key: "35f74addbf6c9ca41f053415cf23a457"
server_key_iv: "51aede2130074e66ab8b1d602881f3f5"
server_key_version: "encryptionkey@0"
I20260320 02:58:53.591286 31883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:53.624423 31883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:53.628852 31883 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:53.629042 31883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:53.630462 31883 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:53.632201 31883 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:53.632257 31883 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.632300 31883 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:53.632319 31883 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.647970 31883 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:46461
I20260320 02:58:53.647995 32011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:46461 every 8 connection(s)
I20260320 02:58:53.650014 31883 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:58:53.652097 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 31883
I20260320 02:58:53.652261 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 31222
I20260320 02:58:53.661515 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:36043
--local_ip_for_outbound_sockets=127.23.132.67
--tserver_master_addrs=127.23.132.126:42639
--webserver_port=45435
--webserver_interface=127.23.132.67
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609 with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.124@KRBTEST.COM
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:58:53.670478 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.652155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:54505 (local address 127.23.132.125:34787)
0320 02:58:53.652409 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.652413 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.655059 (+ 2646us) server_negotiation.cc:408] Connection header received
0320 02:58:53.655482 (+ 423us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.655487 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.655555 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.655632 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.659186 (+ 3554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.659978 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.661248 (+ 1270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.661448 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.664671 (+ 3223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.664696 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.664699 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.664731 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.666953 (+ 2222us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.667609 (+ 656us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.667613 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.667614 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.667664 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.668268 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.668272 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.668274 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.668479 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.668655 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.670158 (+ 1503us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.670299 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260320 02:58:53.670478 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.652026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:42037 (local address 127.23.132.126:42639)
0320 02:58:53.652298 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.652302 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.655565 (+ 3263us) server_negotiation.cc:408] Connection header received
0320 02:58:53.655931 (+ 366us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.655935 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.655993 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.656044 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.659186 (+ 3142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.660033 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.661547 (+ 1514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.661744 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.664774 (+ 3030us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.664795 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.664799 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.664828 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.667001 (+ 2173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.667917 (+ 916us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.667921 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.667923 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.667992 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.669164 (+ 1172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.669168 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.669170 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.669362 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.669463 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.670149 (+ 686us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.670298 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260320 02:58:53.671159 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.653518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:36215 (local address 127.23.132.124:42609)
0320 02:58:53.653854 (+ 336us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.653859 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.655643 (+ 1784us) server_negotiation.cc:408] Connection header received
0320 02:58:53.655941 (+ 298us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.655945 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.656002 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.656056 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.658571 (+ 2515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.659404 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.660306 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.660485 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.664772 (+ 4287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.664812 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.664816 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.664875 (+ 59us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.667427 (+ 2552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.668352 (+ 925us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.668356 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.668358 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.668410 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.668812 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.668815 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.668818 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.669015 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.669100 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.670878 (+ 1778us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.671007 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260320 02:58:53.672154 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.653240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.23.132.66:36215)
0320 02:58:53.655568 (+ 2328us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.655580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.655830 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.656195 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.656198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.657487 (+ 1289us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.658384 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.658403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.659569 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.659572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.660167 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.660177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.660799 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.661476 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.661512 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.664120 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.667680 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.667685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.667687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.668227 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.668628 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.668632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.668634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.668697 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.669525 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.669528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.669655 (+ 127us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.671393 (+ 1738us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.671539 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2263,"spinlock_wait_cycles":300672,"thread_start_us":75,"threads_started":1}
I20260320 02:58:53.672181 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.652188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:42037)
0320 02:58:53.655461 (+ 3273us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.655477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.655798 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.656167 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.656170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.657471 (+ 1301us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.658469 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.658480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.660151 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.660154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.661091 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.661421 (+ 330us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.662324 (+ 903us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.662753 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.662768 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.664621 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.667577 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.667580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.667582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.667810 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.668701 (+ 891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.668704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.668706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.668845 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.669509 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.669513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.669647 (+ 134us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.670085 (+ 438us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.670381 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3067,"mutex_wait_us":27,"spinlock_wait_cycles":44288,"thread_start_us":131,"threads_started":1}
I20260320 02:58:53.672181 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.652188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.23.132.66:54505)
0320 02:58:53.654103 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.654156 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.655139 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.656056 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.656068 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.657471 (+ 1403us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.658469 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.658480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.660126 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.660130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.661091 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.661103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.661573 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.662284 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.662301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.664514 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.667115 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.667122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.667136 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.667497 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.667766 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.667769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.667771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.667908 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.668653 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.668659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.669647 (+ 988us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.670719 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.670931 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"spinlock_wait_cycles":8320,"thread_start_us":133,"threads_started":1}
I20260320 02:58:53.673662 32014 heartbeater.cc:344] Connected to a master server at 127.23.132.125:34787
I20260320 02:58:53.673750 32012 heartbeater.cc:344] Connected to a master server at 127.23.132.124:42609
I20260320 02:58:53.673758 32013 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42639
I20260320 02:58:53.674285 32013 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.674173 32012 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.675364 32013 heartbeater.cc:507] Master 127.23.132.126:42639 requested a full tablet report, sending...
I20260320 02:58:53.675429 32012 heartbeater.cc:507] Master 127.23.132.124:42609 requested a full tablet report, sending...
I20260320 02:58:53.674173 32014 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.676206 32014 heartbeater.cc:507] Master 127.23.132.125:34787 requested a full tablet report, sending...
I20260320 02:58:53.676883 30886 ts_manager.cc:194] Re-registered known tserver with Master: 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461)
I20260320 02:58:53.677176 31500 ts_manager.cc:194] Registered new tserver with Master: 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461)
I20260320 02:58:53.677403 31683 ts_manager.cc:194] Registered new tserver with Master: 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461)
I20260320 02:58:53.677646 30886 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:42037
W20260320 02:58:53.798882 32019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:58:53.799129 32019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:58:53.799185 32019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:58:53.803720 32019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:58:53.804116 32019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:58:53.804296 32019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:58:53.809296 32019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:36043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=45435
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:42639,127.23.132.125:34787,127.23.132.124:42609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:58:53.811518 32019 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:58:53.812546 32019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:58:53.819871 32027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:53.819967 32024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:58:53.819967 32025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:58:53.820272 32019 server_base.cc:1061] running on GCE node
I20260320 02:58:53.820986 32019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:58:53.821587 32019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:58:53.823217 32019 hybrid_clock.cc:648] HybridClock initialized: now 1773975533823177 us; error 51 us; skew 500 ppm
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:58:53.827425 32019 init.cc:377] Logged in from keytab as oryx/127.23.132.67@KRBTEST.COM (short username oryx)
I20260320 02:58:53.828732 32019 webserver.cc:492] Webserver started at http://127.23.132.67:45435/ using document root <none> and password file <none>
I20260320 02:58:53.829341 32019 fs_manager.cc:362] Metadata directory not provided
I20260320 02:58:53.829406 32019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:58:53.834275 32019 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:53.836669 32034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.838275 32019 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:53.838629 32019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "ac75f983b52d436b89f01d48d87119bd"
format_stamp: "Formatted at 2026-03-20 02:58:51 on dist-test-slave-v3hj"
server_key: "2e58339da37c9199a25d84b72e2df9ed"
server_key_iv: "aa3c12943a354afb6f2a34025e4d1710"
server_key_version: "encryptionkey@0"
I20260320 02:58:53.839547 32019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:58:53.869055 32019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:58:53.872751 32019 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:58:53.873051 32019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:58:53.874099 32019 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:58:53.875406 32019 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:58:53.875469 32019 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.875516 32019 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:58:53.875546 32019 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:58:53.891724 32019 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:36043
I20260320 02:58:53.891788 32147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:36043 every 8 connection(s)
I20260320 02:58:53.894109 32019 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:58:53.903935 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 32019
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.125@KRBTEST.COM
Mar 20 02:58:53 dist-test-slave-v3hj krb5kdc[30838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975533, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.124@KRBTEST.COM
I20260320 02:58:53.913376 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.899224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:50693 (local address 127.23.132.126:42639)
0320 02:58:53.899414 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.899417 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.899431 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:58:53.900302 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.900305 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.900367 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.900420 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.902419 (+ 1999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.902941 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.903842 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.904107 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.907675 (+ 3568us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.907693 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.907696 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.907728 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.911174 (+ 3446us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.911700 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.911704 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.911706 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.911761 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.912037 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.912041 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.912042 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.912255 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.912326 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.913060 (+ 734us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.913202 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260320 02:58:53.914346 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.897048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:55695 (local address 127.23.132.125:34787)
0320 02:58:53.897276 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.897281 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.898293 (+ 1012us) server_negotiation.cc:408] Connection header received
0320 02:58:53.900694 (+ 2401us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.900699 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.900828 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.900888 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.903090 (+ 2202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.903825 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.905041 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.905224 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.908129 (+ 2905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.908150 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.908153 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.908184 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.910488 (+ 2304us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.911182 (+ 694us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.911186 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.911187 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.911235 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.911582 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.911585 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.911587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.912022 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.912166 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.913939 (+ 1773us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.914087 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260320 02:58:53.914530 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.897278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:47359 (local address 127.23.132.124:42609)
0320 02:58:53.897439 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.897443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.898292 (+ 849us) server_negotiation.cc:408] Connection header received
0320 02:58:53.900430 (+ 2138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.900434 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.900485 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.900561 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.903972 (+ 3411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.905171 (+ 1199us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.906708 (+ 1537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.906903 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.910094 (+ 3191us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.910130 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.910134 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.910181 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.912275 (+ 2094us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.912784 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.912788 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.912789 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.912837 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.913090 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.913093 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.913094 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.913265 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.913347 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.914274 (+ 927us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.914383 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260320 02:58:53.915356 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.897357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.125:34787 (local address 127.23.132.67:55695)
0320 02:58:53.898138 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.898184 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.900297 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.901031 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.901034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.901385 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.902259 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.902275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.904258 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.904262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.904849 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.904860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.905379 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.906192 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.906208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.907968 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.910658 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.910666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.910681 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.911055 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.911329 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.911332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.911334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.911479 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.912265 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.912274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.912712 (+ 438us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.913032 (+ 320us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.913444 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"spinlock_wait_cycles":79104,"thread_start_us":169,"threads_started":1}
I20260320 02:58:53.915349 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.898752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:50693)
0320 02:58:53.899095 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.899112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.900046 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.900751 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.900760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.901382 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.902258 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.902273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.903064 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.903068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.903694 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.903704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.903946 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.904713 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.904752 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.907494 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.911315 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.911319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.911321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.911571 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.911873 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.911876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.911879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.911928 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.912397 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.912401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.912710 (+ 309us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.913858 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.914044 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"spinlock_wait_cycles":5504,"thread_start_us":87,"threads_started":1}
I20260320 02:58:53.916839 32150 heartbeater.cc:344] Connected to a master server at 127.23.132.125:34787
I20260320 02:58:53.915337 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.897359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.124:42609 (local address 127.23.132.67:47359)
0320 02:58:53.898193 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.898209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.900032 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.900751 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.900761 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.901383 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:58:53.902259 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.902275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.905314 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.905318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.906174 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.906185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.906304 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.907309 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:58:53.907326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:58:53.909689 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:58:53.912398 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.912401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.912403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.912594 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.912940 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:58:53.912943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:58:53.912945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:58:53.912998 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:58:53.913392 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:58:53.913395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:58:53.914087 (+ 692us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.914179 (+ 92us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.914313 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":202,"spinlock_wait_cycles":626176,"thread_start_us":297,"threads_started":1}
I20260320 02:58:53.917119 32150 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.917711 32149 heartbeater.cc:344] Connected to a master server at 127.23.132.126:42639
I20260320 02:58:53.917842 32150 heartbeater.cc:507] Master 127.23.132.125:34787 requested a full tablet report, sending...
I20260320 02:58:53.918089 32149 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.918310 32149 heartbeater.cc:507] Master 127.23.132.126:42639 requested a full tablet report, sending...
I20260320 02:58:53.918032 32148 heartbeater.cc:344] Connected to a master server at 127.23.132.124:42609
I20260320 02:58:53.918799 32148 heartbeater.cc:461] Registering TS with master...
I20260320 02:58:53.919245 32148 heartbeater.cc:507] Master 127.23.132.124:42609 requested a full tablet report, sending...
I20260320 02:58:53.919946 30886 ts_manager.cc:194] Re-registered known tserver with Master: ac75f983b52d436b89f01d48d87119bd (127.23.132.67:36043)
I20260320 02:58:53.920056 31500 ts_manager.cc:194] Registered new tserver with Master: ac75f983b52d436b89f01d48d87119bd (127.23.132.67:36043)
I20260320 02:58:53.920083 31683 ts_manager.cc:194] Registered new tserver with Master: ac75f983b52d436b89f01d48d87119bd (127.23.132.67:36043)
I20260320 02:58:53.920634 30886 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:50693
I20260320 02:58:53.923709 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.905965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.23.132.125:34787)
0320 02:58:53.909972 (+ 4007us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.909977 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.909995 (+ 18us) server_negotiation.cc:408] Connection header received
0320 02:58:53.910039 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.910043 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.910097 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.910194 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.911336 (+ 1142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.912226 (+ 890us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.913625 (+ 1399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.913851 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.915202 (+ 1351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.915224 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.915228 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.915256 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.921403 (+ 6147us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.921881 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.921885 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.921886 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.921966 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.922210 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.922213 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.922215 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.922408 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.922538 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.923283 (+ 745us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.923480 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3809,"thread_start_us":74,"threads_started":1}
I20260320 02:58:53.928963 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.907188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.23.132.126:42639)
0320 02:58:53.913590 (+ 6402us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.913595 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.913606 (+ 11us) server_negotiation.cc:408] Connection header received
0320 02:58:53.913640 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.913643 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.913691 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.913770 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.916856 (+ 3086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.917617 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.918617 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.918808 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.920928 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.920944 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.920948 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.920972 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.926032 (+ 5060us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.927126 (+ 1094us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.927130 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.927132 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.927193 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.927810 (+ 617us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.927813 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.927815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.928132 (+ 317us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.928239 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.928551 (+ 312us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.928764 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6334,"thread_start_us":77,"threads_started":1}
I20260320 02:58:53.929595 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.907473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.23.132.124:42609)
0320 02:58:53.908604 (+ 1131us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.908608 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.916576 (+ 7968us) server_negotiation.cc:408] Connection header received
0320 02:58:53.916700 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.916704 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.916754 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.916832 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:53.917908 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.921992 (+ 4084us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.923599 (+ 1607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.923784 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.924595 (+ 811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:53.924617 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:53.924620 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:53.924647 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:53.927096 (+ 2449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.927943 (+ 847us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.927948 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.927949 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.928001 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:53.928456 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:53.928461 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:53.928463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:53.928650 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:53.928780 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.929372 (+ 592us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.929479 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":61,"threads_started":1}
I20260320 02:58:53.932843 30886 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:58:53.935809 30886 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:58:53.958549 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.953257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:46461 (local address 127.0.0.1:35474)
0320 02:58:53.954567 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.954581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.954680 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.955193 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.955196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.955213 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.955489 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.955499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.957258 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.957261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.958133 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.958143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.958248 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.958272 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.958328 (+ 56us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.958396 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"thread_start_us":122,"threads_started":1}
I20260320 02:58:53.959136 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.953394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.23.132.66:46461)
0320 02:58:53.954122 (+ 728us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.954128 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.954654 (+ 526us) server_negotiation.cc:408] Connection header received
0320 02:58:53.954953 (+ 299us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.954959 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.955075 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.955195 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.955589 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.956675 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.958238 (+ 1563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.958773 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.958900 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.958958 (+ 58us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.959037 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":119,"threads_started":1}
I20260320 02:58:53.960322 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.952608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40327 (local address 127.0.0.1:58356)
0320 02:58:53.954323 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.954341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.954456 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.955418 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.955423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.955446 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.955817 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.955825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.957453 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.957457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.959941 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.959950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.960057 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.960077 (+ 20us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.960122 (+ 45us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.960193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"spinlock_wait_cycles":11264,"thread_start_us":154,"threads_started":1}
I20260320 02:58:53.960588 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.953594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:36043 (local address 127.0.0.1:55500)
0320 02:58:53.954928 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:53.954938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:53.955013 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:53.955443 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:53.955446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:53.955478 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:53.955792 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.955804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.957956 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.957964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:53.960179 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:53.960191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.960320 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.960349 (+ 29us) client_negotiation.cc:769] Sending connection context
0320 02:58:53.960398 (+ 49us) client_negotiation.cc:240] Negotiation successful
0320 02:58:53.960466 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20260320 02:58:53.961540 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.954414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.23.132.67:36043)
0320 02:58:53.954801 (+ 387us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.954807 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.955043 (+ 236us) server_negotiation.cc:408] Connection header received
0320 02:58:53.955124 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.955127 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.955273 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.955409 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.956063 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.957156 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.960822 (+ 3666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.961222 (+ 400us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.961302 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.961354 (+ 52us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.961419 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":123,"threads_started":1}
I20260320 02:58:53.962050 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:53.953150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.23.132.65:40327)
0320 02:58:53.953754 (+ 604us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:53.953761 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:53.954561 (+ 800us) server_negotiation.cc:408] Connection header received
0320 02:58:53.954802 (+ 241us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:53.954809 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:53.955033 (+ 224us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:53.955191 (+ 158us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:53.956023 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.957339 (+ 1316us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:53.960073 (+ 2734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:53.960598 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:53.961713 (+ 1115us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:53.961773 (+ 60us) server_negotiation.cc:300] Negotiation successful
0320 02:58:53.961896 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":95,"threads_started":1}
I20260320 02:58:53.962795 31946 tablet_service.cc:1511] Processing CreateTablet for tablet 43b255b32a5b422fb4a2958bcaa6dfac (DEFAULT_TABLE table=test-table [id=d5ceaafa305e4785a5cf42f7d9257ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:53.963658 31946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b255b32a5b422fb4a2958bcaa6dfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:53.964371 31810 tablet_service.cc:1511] Processing CreateTablet for tablet 43b255b32a5b422fb4a2958bcaa6dfac (DEFAULT_TABLE table=test-table [id=d5ceaafa305e4785a5cf42f7d9257ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:53.965245 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b255b32a5b422fb4a2958bcaa6dfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:53.972393 32179 tablet_bootstrap.cc:492] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Bootstrap starting.
I20260320 02:58:53.972487 32082 tablet_service.cc:1511] Processing CreateTablet for tablet 43b255b32a5b422fb4a2958bcaa6dfac (DEFAULT_TABLE table=test-table [id=d5ceaafa305e4785a5cf42f7d9257ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:53.973443 32082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b255b32a5b422fb4a2958bcaa6dfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:53.975193 32179 tablet_bootstrap.cc:654] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:53.975872 32179 log.cc:826] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:53.978199 32180 tablet_bootstrap.cc:492] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Bootstrap starting.
I20260320 02:58:53.979660 32179 tablet_bootstrap.cc:492] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: No bootstrap required, opened a new log
I20260320 02:58:53.980136 32179 ts_tablet_manager.cc:1403] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260320 02:58:53.981510 32180 tablet_bootstrap.cc:654] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:53.982440 32180 log.cc:826] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:53.983353 32179 raft_consensus.cc:359] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:53.983592 32179 raft_consensus.cc:385] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:53.983654 32179 raft_consensus.cc:740] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afdd0e581c244e6a0a63ca52c94a592, State: Initialized, Role: FOLLOWER
I20260320 02:58:53.984089 32179 consensus_queue.cc:260] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:53.985042 32179 ts_tablet_manager.cc:1434] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260320 02:58:53.985203 31877 heartbeater.cc:499] Master 127.23.132.126:42639 was elected leader, sending a full tablet report...
I20260320 02:58:53.985736 32180 tablet_bootstrap.cc:492] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: No bootstrap required, opened a new log
I20260320 02:58:53.986483 32180 ts_tablet_manager.cc:1403] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260320 02:58:53.988869 32184 tablet_bootstrap.cc:492] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Bootstrap starting.
I20260320 02:58:53.991276 32180 raft_consensus.cc:359] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:53.991519 32180 raft_consensus.cc:385] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:53.991577 32180 raft_consensus.cc:740] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf0ee39c16f4da6ad7a02c9c06fbaf3, State: Initialized, Role: FOLLOWER
I20260320 02:58:53.992838 32180 consensus_queue.cc:260] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:53.994335 32013 heartbeater.cc:499] Master 127.23.132.126:42639 was elected leader, sending a full tablet report...
I20260320 02:58:53.994590 32180 ts_tablet_manager.cc:1434] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260320 02:58:53.995077 32184 tablet_bootstrap.cc:654] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:53.998504 32184 log.cc:826] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Log is configured to *not* fsync() on all Append() calls
I20260320 02:58:54.004285 32184 tablet_bootstrap.cc:492] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: No bootstrap required, opened a new log
I20260320 02:58:54.004693 32184 ts_tablet_manager.cc:1403] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260320 02:58:54.008916 32184 raft_consensus.cc:359] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:54.009179 32184 raft_consensus.cc:385] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:54.009236 32184 raft_consensus.cc:740] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac75f983b52d436b89f01d48d87119bd, State: Initialized, Role: FOLLOWER
I20260320 02:58:54.009867 32184 consensus_queue.cc:260] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:54.011277 32149 heartbeater.cc:499] Master 127.23.132.126:42639 was elected leader, sending a full tablet report...
I20260320 02:58:54.011489 32184 ts_tablet_manager.cc:1434] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260320 02:58:54.136420 31879 tablet.cc:2379] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:58:54.146631 32151 tablet.cc:2379] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:58:54.152240 32015 tablet.cc:2379] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:58:54.211465 32187 raft_consensus.cc:493] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:54.211704 32187 raft_consensus.cc:515] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:54.213131 32187 leader_election.cc:290] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327), 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461)
I20260320 02:58:54.217861 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.213397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:46461 (local address 127.23.132.67:52875)
0320 02:58:54.213526 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:54.213541 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:54.213646 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:54.214029 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:54.214032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:54.214050 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:54.214422 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:54.214431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.216409 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.216414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:54.217382 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:54.217395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.217527 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.217551 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:58:54.217601 (+ 50us) client_negotiation.cc:240] Negotiation successful
0320 02:58:54.217669 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260320 02:58:54.217864 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.213538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40327 (local address 127.23.132.67:37943)
0320 02:58:54.213712 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:54.213725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:54.213781 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:54.214346 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:54.214349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:54.214367 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:54.214579 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:54.214585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.216409 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.216414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:54.217382 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:54.217395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.217527 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.217551 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:58:54.217601 (+ 50us) client_negotiation.cc:240] Negotiation successful
0320 02:58:54.217669 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260320 02:58:54.218423 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.213539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:52875 (local address 127.23.132.66:46461)
0320 02:58:54.213676 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:54.213681 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:54.213697 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:58:54.213756 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:54.213760 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:54.213905 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:54.213999 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:54.214690 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.216240 (+ 1550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.217554 (+ 1314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.218147 (+ 593us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.218189 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:54.218245 (+ 56us) server_negotiation.cc:300] Negotiation successful
0320 02:58:54.218308 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260320 02:58:54.218423 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.213839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:37943 (local address 127.23.132.65:40327)
0320 02:58:54.214055 (+ 216us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:54.214060 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:54.214072 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:58:54.214129 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:54.214131 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:54.214228 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:54.214306 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:54.214756 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.216266 (+ 1510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.217554 (+ 1288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.218175 (+ 621us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.218209 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:54.218252 (+ 43us) server_negotiation.cc:300] Negotiation successful
0320 02:58:54.218308 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260320 02:58:54.219173 31966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b255b32a5b422fb4a2958bcaa6dfac" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" is_pre_election: true
I20260320 02:58:54.219175 31830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b255b32a5b422fb4a2958bcaa6dfac" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afdd0e581c244e6a0a63ca52c94a592" is_pre_election: true
I20260320 02:58:54.219488 31830 raft_consensus.cc:2468] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 0.
I20260320 02:58:54.219488 31966 raft_consensus.cc:2468] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 0.
I20260320 02:58:54.220345 32036 leader_election.cc:304] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afdd0e581c244e6a0a63ca52c94a592, ac75f983b52d436b89f01d48d87119bd; no voters:
I20260320 02:58:54.220686 32187 raft_consensus.cc:2804] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:58:54.220784 32187 raft_consensus.cc:493] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:54.220829 32187 raft_consensus.cc:3060] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:54.222163 32187 raft_consensus.cc:515] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:54.222543 32187 leader_election.cc:290] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 election: Requested vote from peers 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327), 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461)
I20260320 02:58:54.223116 31966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b255b32a5b422fb4a2958bcaa6dfac" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3"
I20260320 02:58:54.223116 31830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b255b32a5b422fb4a2958bcaa6dfac" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afdd0e581c244e6a0a63ca52c94a592"
I20260320 02:58:54.223243 31966 raft_consensus.cc:3060] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:54.223244 31830 raft_consensus.cc:3060] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:54.224403 31966 raft_consensus.cc:2468] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 1.
I20260320 02:58:54.224404 31830 raft_consensus.cc:2468] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 1.
I20260320 02:58:54.224781 32038 leader_election.cc:304] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf0ee39c16f4da6ad7a02c9c06fbaf3, ac75f983b52d436b89f01d48d87119bd; no voters:
I20260320 02:58:54.225098 32187 raft_consensus.cc:2804] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:54.225777 32187 raft_consensus.cc:697] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 1 LEADER]: Becoming Leader. State: Replica: ac75f983b52d436b89f01d48d87119bd, State: Running, Role: LEADER
I20260320 02:58:54.226293 32187 consensus_queue.cc:237] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } }
I20260320 02:58:54.230171 30886 catalog_manager.cc:5671] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac75f983b52d436b89f01d48d87119bd (127.23.132.67). New cstate: current_term: 1 leader_uuid: "ac75f983b52d436b89f01d48d87119bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } health_report { overall_health: HEALTHY } } }
I20260320 02:58:54.262878 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.257263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.23.132.67:36043)
0320 02:58:54.257428 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:54.257433 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:54.257511 (+ 78us) server_negotiation.cc:408] Connection header received
0320 02:58:54.257817 (+ 306us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:54.257822 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:54.257887 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:54.257989 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:58:54.258359 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.259249 (+ 890us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.261138 (+ 1889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.261380 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.261511 (+ 131us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:58:54.262164 (+ 653us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:58:54.262288 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:54.262652 (+ 364us) server_negotiation.cc:300] Negotiation successful
0320 02:58:54.262717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260320 02:58:54.272286 31830 raft_consensus.cc:1275] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Refusing update from remote peer ac75f983b52d436b89f01d48d87119bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:54.273278 31966 raft_consensus.cc:1275] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Refusing update from remote peer ac75f983b52d436b89f01d48d87119bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:54.275017 32188 consensus_queue.cc:1048] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260320 02:58:54.275660 32188 consensus_queue.cc:1048] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260320 02:58:54.290243 32191 mvcc.cc:204] Tried to move back new op lower bound from 7266203788365574144 to 7266203788193738752. Current Snapshot: MvccSnapshot[applied={T|T < 7266203788365574144}]
I20260320 02:58:54.292250 32192 mvcc.cc:204] Tried to move back new op lower bound from 7266203788365574144 to 7266203788193738752. Current Snapshot: MvccSnapshot[applied={T|T < 7266203788365574144}]
I20260320 02:58:54.295097 32193 mvcc.cc:204] Tried to move back new op lower bound from 7266203788365574144 to 7266203788193738752. Current Snapshot: MvccSnapshot[applied={T|T < 7266203788365574144}]
I20260320 02:58:54.318020 30880 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:54.318199 30880 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:54.326386 30880 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a391b242632f4333b94724207e5d6889: Sending DeleteTablet for 3 replicas of tablet 43b255b32a5b422fb4a2958bcaa6dfac
I20260320 02:58:54.327164 31810 tablet_service.cc:1558] Processing DeleteTablet for tablet 43b255b32a5b422fb4a2958bcaa6dfac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:54 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:58356
I20260320 02:58:54.327160 31946 tablet_service.cc:1558] Processing DeleteTablet for tablet 43b255b32a5b422fb4a2958bcaa6dfac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:54 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:35474
I20260320 02:58:54.327539 32082 tablet_service.cc:1558] Processing DeleteTablet for tablet 43b255b32a5b422fb4a2958bcaa6dfac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:54 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:55500
I20260320 02:58:54.328028 32203 tablet_replica.cc:333] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: stopping tablet replica
I20260320 02:58:54.328366 32203 raft_consensus.cc:2243] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:58:54.329025 32203 raft_consensus.cc:2272] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:54.330770 32203 ts_tablet_manager.cc:1916] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:54.331569 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 30854
I20260320 02:58:54.335103 32203 ts_tablet_manager.cc:1929] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:54.335515 32203 log.cc:1199] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/43b255b32a5b422fb4a2958bcaa6dfac
I20260320 02:58:54.336066 32203 ts_tablet_manager.cc:1950] T 43b255b32a5b422fb4a2958bcaa6dfac P ac75f983b52d436b89f01d48d87119bd: Deleting consensus metadata
I20260320 02:58:54.338244 32202 tablet_replica.cc:333] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: stopping tablet replica
I20260320 02:58:54.338640 32202 raft_consensus.cc:2243] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:54.339124 32202 raft_consensus.cc:2272] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:54.340749 32202 ts_tablet_manager.cc:1916] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260320 02:58:54.340986 32035 connection.cc:537] server connection from 127.0.0.1:55500 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260320 02:58:54.344040 32201 tablet_replica.cc:333] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: stopping tablet replica
I20260320 02:58:54.344350 32201 raft_consensus.cc:2243] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:54.346427 32202 ts_tablet_manager.cc:1929] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:54.346750 32202 log.cc:1199] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/43b255b32a5b422fb4a2958bcaa6dfac
I20260320 02:58:54.347358 32202 ts_tablet_manager.cc:1950] T 43b255b32a5b422fb4a2958bcaa6dfac P 3afdd0e581c244e6a0a63ca52c94a592: Deleting consensus metadata
I20260320 02:58:54.347699 32201 raft_consensus.cc:2272] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260320 02:58:54.349695 31765 connection.cc:441] server connection from 127.0.0.1:58356 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58356 (request call id 1) could send its response
I20260320 02:58:54.351025 32201 ts_tablet_manager.cc:1916] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:54.354885 32201 ts_tablet_manager.cc:1929] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:54.355163 32201 log.cc:1199] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/43b255b32a5b422fb4a2958bcaa6dfac
I20260320 02:58:54.355597 32201 ts_tablet_manager.cc:1950] T 43b255b32a5b422fb4a2958bcaa6dfac P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Deleting consensus metadata
W20260320 02:58:54.357357 31899 connection.cc:441] server connection from 127.0.0.1:35474 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35474 (request call id 1) could send its response
I20260320 02:58:54.367852 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.356216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.23.132.124:42609)
0320 02:58:54.356393 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:54.356399 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:54.357283 (+ 884us) server_negotiation.cc:408] Connection header received
0320 02:58:54.357476 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:54.357480 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:54.357540 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:54.357593 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:54.358743 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.359241 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.359863 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.360053 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.362482 (+ 2429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:54.362504 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:54.362506 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:54.362534 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:54.364211 (+ 1677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:54.364571 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:54.364575 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:54.364576 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:54.364617 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:54.365467 (+ 850us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:54.365481 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:54.365483 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:54.365781 (+ 298us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:54.366186 (+ 405us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:54.367387 (+ 1201us) server_negotiation.cc:300] Negotiation successful
0320 02:58:54.367542 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260320 02:58:54.374615 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:54.348528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.23.132.125:34787)
0320 02:58:54.348696 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:54.348701 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:54.354230 (+ 5529us) server_negotiation.cc:408] Connection header received
0320 02:58:54.354425 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:54.354430 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:54.354616 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:54.354737 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:58:54.362967 (+ 8230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.363748 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:54.365037 (+ 1289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:54.365196 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:54.366762 (+ 1566us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:58:54.366788 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:58:54.366793 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:58:54.366824 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:58:54.369721 (+ 2897us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:54.370898 (+ 1177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:54.370903 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:54.370904 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:54.371091 (+ 187us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:58:54.371445 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:58:54.371447 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:58:54.371449 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:58:54.371759 (+ 310us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:58:54.371997 (+ 238us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:54.373430 (+ 1433us) server_negotiation.cc:300] Negotiation successful
0320 02:58:54.373579 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260320 02:58:55.296880 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.296450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:56351)
0320 02:58:55.296697 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.296779 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
W20260320 02:58:55.297294 32013 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:42639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.299211 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.298783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:43823)
0320 02:58:55.299060 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.299124 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260320 02:58:55.299510 31877 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:42639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.305696 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.305226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:47291)
0320 02:58:55.305510 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.305581 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
W20260320 02:58:55.306239 32149 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:42639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.825981 32214 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a391b242632f4333b94724207e5d6889)
I20260320 02:58:55.826203 32214 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:55.826789 32215 raft_consensus.cc:493] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a391b242632f4333b94724207e5d6889)
I20260320 02:58:55.826926 32215 raft_consensus.cc:515] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:55.827395 32214 leader_election.cc:290] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a391b242632f4333b94724207e5d6889 (127.23.132.126:42639), 33519057c23f4e96b2e9456707a75b3d (127.23.132.125:34787)
I20260320 02:58:55.827914 31510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8221429cd61248029bfa740ab799e3f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33519057c23f4e96b2e9456707a75b3d" is_pre_election: true
I20260320 02:58:55.828078 31510 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221429cd61248029bfa740ab799e3f6 in term 1.
I20260320 02:58:55.828080 32215 leader_election.cc:290] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a391b242632f4333b94724207e5d6889 (127.23.132.126:42639), 8221429cd61248029bfa740ab799e3f6 (127.23.132.124:42609)
I20260320 02:58:55.828003 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.827555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57472)
0320 02:58:55.827816 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.827891 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":53,"threads_started":1}
I20260320 02:58:55.828804 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.828164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57482)
0320 02:58:55.828563 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.828705 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":55,"threads_started":1}
I20260320 02:58:55.828819 31693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33519057c23f4e96b2e9456707a75b3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8221429cd61248029bfa740ab799e3f6" is_pre_election: true
I20260320 02:58:55.829034 31693 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33519057c23f4e96b2e9456707a75b3d in term 1.
W20260320 02:58:55.829234 31667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.828819 31670 leader_election.cc:304] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33519057c23f4e96b2e9456707a75b3d, 8221429cd61248029bfa740ab799e3f6; no voters:
I20260320 02:58:55.829571 32214 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260320 02:58:55.829691 32214 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a391b242632f4333b94724207e5d6889)
I20260320 02:58:55.829746 32214 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 1 FOLLOWER]: Advancing to term 2
W20260320 02:58:55.829955 31484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.830650 31484 leader_election.cc:304] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33519057c23f4e96b2e9456707a75b3d, 8221429cd61248029bfa740ab799e3f6; no voters:
I20260320 02:58:55.830819 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.830552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57496)
0320 02:58:55.830704 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.830755 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260320 02:58:55.831243 32215 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260320 02:58:55.831236 31667 leader_election.cc:336] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.831316 32215 raft_consensus.cc:493] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a391b242632f4333b94724207e5d6889)
I20260320 02:58:55.831351 32215 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 1 FOLLOWER]: Advancing to term 2
I20260320 02:58:55.831876 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.831383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57512)
0320 02:58:55.831723 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.831800 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260320 02:58:55.832029 31484 leader_election.cc:336] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.832255 32214 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:55.833014 32214 leader_election.cc:290] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 2 election: Requested vote from peers a391b242632f4333b94724207e5d6889 (127.23.132.126:42639), 33519057c23f4e96b2e9456707a75b3d (127.23.132.125:34787)
I20260320 02:58:55.833050 32215 raft_consensus.cc:515] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:55.833454 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.832974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57526)
0320 02:58:55.833121 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.833382 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260320 02:58:55.833669 32215 leader_election.cc:290] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [CANDIDATE]: Term 2 election: Requested vote from peers a391b242632f4333b94724207e5d6889 (127.23.132.126:42639), 8221429cd61248029bfa740ab799e3f6 (127.23.132.124:42609)
I20260320 02:58:55.833611 31510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8221429cd61248029bfa740ab799e3f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33519057c23f4e96b2e9456707a75b3d"
I20260320 02:58:55.833911 31510 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8221429cd61248029bfa740ab799e3f6 in current term 2: Already voted for candidate 33519057c23f4e96b2e9456707a75b3d in this term.
I20260320 02:58:55.834113 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.833844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57534)
0320 02:58:55.833999 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.834051 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260320 02:58:55.834770 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.834054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57540)
0320 02:58:55.834153 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.834202 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260320 02:58:55.834740 31693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33519057c23f4e96b2e9456707a75b3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8221429cd61248029bfa740ab799e3f6"
I20260320 02:58:55.835119 31693 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33519057c23f4e96b2e9456707a75b3d in current term 2: Already voted for candidate 8221429cd61248029bfa740ab799e3f6 in this term.
W20260320 02:58:55.835165 31667 leader_election.cc:336] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.835559 31667 leader_election.cc:304] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8221429cd61248029bfa740ab799e3f6; no voters: 33519057c23f4e96b2e9456707a75b3d, a391b242632f4333b94724207e5d6889
I20260320 02:58:55.835594 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:55.834839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57542)
0320 02:58:55.835154 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:55.835333 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
W20260320 02:58:55.835737 31484 leader_election.cc:336] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:55.836113 32214 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260320 02:58:55.836434 31484 leader_election.cc:304] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33519057c23f4e96b2e9456707a75b3d; no voters: 8221429cd61248029bfa740ab799e3f6, a391b242632f4333b94724207e5d6889
I20260320 02:58:55.837023 32215 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260320 02:58:56.298800 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:56.298219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:36827)
0320 02:58:56.298508 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:56.298577 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
I20260320 02:58:56.300567 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:56.300173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:48943)
0320 02:58:56.300427 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:56.300476 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260320 02:58:56.307483 32222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:56.306907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:33839)
0320 02:58:56.307312 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:56.307383 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":166,"threads_started":1}
I20260320 02:58:57.300299 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.299758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:34797)
0320 02:58:57.300099 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.300167 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260320 02:58:57.302243 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.301454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:42733)
0320 02:58:57.301918 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.302115 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":205,"threads_started":1}
I20260320 02:58:57.308748 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.308281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:54677)
0320 02:58:57.308540 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.308597 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260320 02:58:57.573853 32226 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:57.574064 32226 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:57.574518 32226 leader_election.cc:290] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a391b242632f4333b94724207e5d6889 (127.23.132.126:42639), 33519057c23f4e96b2e9456707a75b3d (127.23.132.125:34787)
I20260320 02:58:57.575259 31510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8221429cd61248029bfa740ab799e3f6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33519057c23f4e96b2e9456707a75b3d" is_pre_election: true
I20260320 02:58:57.575485 31510 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221429cd61248029bfa740ab799e3f6 in term 2.
I20260320 02:58:57.575529 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.574842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57792)
0320 02:58:57.575265 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.575366 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":86,"threads_started":1}
I20260320 02:58:57.576041 31670 leader_election.cc:304] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33519057c23f4e96b2e9456707a75b3d, 8221429cd61248029bfa740ab799e3f6; no voters:
I20260320 02:58:57.576228 32226 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260320 02:58:57.576287 32226 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:57.576314 32226 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 2 FOLLOWER]: Advancing to term 3
I20260320 02:58:57.577179 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.576465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57806)
0320 02:58:57.576958 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.577101 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260320 02:58:57.577328 31667 leader_election.cc:336] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:57.577602 32226 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:57.578051 32226 leader_election.cc:290] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 3 election: Requested vote from peers a391b242632f4333b94724207e5d6889 (127.23.132.126:42639), 33519057c23f4e96b2e9456707a75b3d (127.23.132.125:34787)
I20260320 02:58:57.578437 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.578186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57820)
0320 02:58:57.578332 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.578380 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260320 02:58:57.578711 31510 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8221429cd61248029bfa740ab799e3f6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33519057c23f4e96b2e9456707a75b3d"
I20260320 02:58:57.579001 31510 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 2 FOLLOWER]: Advancing to term 3
I20260320 02:58:57.579428 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.579178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57834)
0320 02:58:57.579325 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.579371 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260320 02:58:57.579640 31667 leader_election.cc:336] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
I20260320 02:58:57.580466 31510 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221429cd61248029bfa740ab799e3f6 in term 3.
I20260320 02:58:57.580788 31670 leader_election.cc:304] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33519057c23f4e96b2e9456707a75b3d, 8221429cd61248029bfa740ab799e3f6; no voters: a391b242632f4333b94724207e5d6889
I20260320 02:58:57.580950 32226 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260320 02:58:57.581174 32226 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [term 3 LEADER]: Becoming Leader. State: Replica: 8221429cd61248029bfa740ab799e3f6, State: Running, Role: LEADER
I20260320 02:58:57.581437 32226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } }
I20260320 02:58:57.582931 32229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8221429cd61248029bfa740ab799e3f6. Latest consensus state: current_term: 3 leader_uuid: "8221429cd61248029bfa740ab799e3f6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:57.583032 32229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:57.583307 32231 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:58:57.585403 32231 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:58:57.585867 32231 catalog_manager.cc:1269] Loaded cluster ID: 0460188e15544f349f52287fff0d88cd
I20260320 02:58:57.585912 32231 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:58:57.586474 32231 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:58:57.587030 32231 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Loaded TSK: 0
I20260320 02:58:57.587721 32231 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:58:57.641150 31683 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:58:57.643764 31683 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:58:57.647296 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.647013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57850)
0320 02:58:57.647157 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.647219 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260320 02:58:57.647450 31510 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [term 3 FOLLOWER]: Refusing update from remote peer 8221429cd61248029bfa740ab799e3f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260320 02:58:57.648140 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.647849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.0.0.1:57856)
0320 02:58:57.648039 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.648088 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260320 02:58:57.648049 32226 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260320 02:58:57.648270 31667 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 -> Peer a391b242632f4333b94724207e5d6889 (127.23.132.126:42639): Couldn't send request to peer a391b242632f4333b94724207e5d6889. Status: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260320 02:58:57.649888 32233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8221429cd61248029bfa740ab799e3f6. Latest consensus state: current_term: 3 leader_uuid: "8221429cd61248029bfa740ab799e3f6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:57.650017 32233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:57.651616 32226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8221429cd61248029bfa740ab799e3f6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:57.651752 32226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6 [sys.catalog]: This master's current role is: LEADER
I20260320 02:58:57.652022 32233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8221429cd61248029bfa740ab799e3f6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a391b242632f4333b94724207e5d6889" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 42639 } } peers { permanent_uuid: "33519057c23f4e96b2e9456707a75b3d" member_type: VOTER last_known_addr { host: "127.23.132.125" port: 34787 } attrs { promote: false } } peers { permanent_uuid: "8221429cd61248029bfa740ab799e3f6" member_type: VOTER last_known_addr { host: "127.23.132.124" port: 42609 } attrs { promote: false } } }
I20260320 02:58:57.652128 32233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33519057c23f4e96b2e9456707a75b3d [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:58:57.664714 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.660774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:36043 (local address 127.0.0.1:55522)
0320 02:58:57.660932 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.660948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:57.661035 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:57.661577 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:57.661582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:57.661623 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:57.662126 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:57.662138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.663422 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.663426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:57.664333 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:57.664346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.664442 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.664464 (+ 22us) client_negotiation.cc:769] Sending connection context
0320 02:58:57.664510 (+ 46us) client_negotiation.cc:240] Negotiation successful
0320 02:58:57.664579 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260320 02:58:57.665179 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.661723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40327 (local address 127.0.0.1:58364)
0320 02:58:57.662281 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.662296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:57.662385 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:57.662742 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:57.662746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:57.662765 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:57.663108 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:57.663117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.664057 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.664061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:57.664846 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:57.664856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.664955 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.664973 (+ 18us) client_negotiation.cc:769] Sending connection context
0320 02:58:57.665015 (+ 42us) client_negotiation.cc:240] Negotiation successful
0320 02:58:57.665064 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":85,"thread_start_us":119,"threads_started":1}
I20260320 02:58:57.665213 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.660930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.23.132.67:36043)
0320 02:58:57.661249 (+ 319us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:57.661253 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:57.661268 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:58:57.661322 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:57.661325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:57.661376 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:57.661475 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:57.662471 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.663316 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.664448 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.664904 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.664948 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:57.665013 (+ 65us) server_negotiation.cc:300] Negotiation successful
0320 02:58:57.665090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
I20260320 02:58:57.665681 32082 tablet_service.cc:1511] Processing CreateTablet for tablet a5bc21c3e9ab4003b697d645e55ef14d (DEFAULT_TABLE table=test-table [id=66b456ce820f433aa91cb9ab28982c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:57.666006 32082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5bc21c3e9ab4003b697d645e55ef14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:57.666155 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.661809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.23.132.65:40327)
0320 02:58:57.662415 (+ 606us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:57.662420 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:57.662439 (+ 19us) server_negotiation.cc:408] Connection header received
0320 02:58:57.662500 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:57.662503 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:57.662561 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:57.662663 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:57.663262 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.663955 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.665378 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.665880 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.665918 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:57.665970 (+ 52us) server_negotiation.cc:300] Negotiation successful
0320 02:58:57.666028 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":97,"threads_started":1}
I20260320 02:58:57.666319 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.661248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:46461 (local address 127.0.0.1:35476)
0320 02:58:57.663487 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.663500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:58:57.663574 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:58:57.663893 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:58:57.663896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:58:57.663916 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:58:57.664155 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:57.664162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.665160 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.665164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:58:57.666007 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:58:57.666016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.666113 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.666127 (+ 14us) client_negotiation.cc:769] Sending connection context
0320 02:58:57.666169 (+ 42us) client_negotiation.cc:240] Negotiation successful
0320 02:58:57.666220 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":170,"threads_started":1}
I20260320 02:58:57.666716 31810 tablet_service.cc:1511] Processing CreateTablet for tablet a5bc21c3e9ab4003b697d645e55ef14d (DEFAULT_TABLE table=test-table [id=66b456ce820f433aa91cb9ab28982c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:57.666996 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5bc21c3e9ab4003b697d645e55ef14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:57.667433 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.661380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.23.132.66:46461)
0320 02:58:57.662650 (+ 1270us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:57.662655 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:57.663669 (+ 1014us) server_negotiation.cc:408] Connection header received
0320 02:58:57.663729 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:57.663733 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:57.663783 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:57.663859 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:58:57.664274 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.665049 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.666457 (+ 1408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.667138 (+ 681us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.667178 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:57.667233 (+ 55us) server_negotiation.cc:300] Negotiation successful
0320 02:58:57.667295 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":75,"threads_started":1}
I20260320 02:58:57.668416 32243 tablet_bootstrap.cc:492] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Bootstrap starting.
I20260320 02:58:57.669384 32244 tablet_bootstrap.cc:492] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592: Bootstrap starting.
I20260320 02:58:57.669600 31946 tablet_service.cc:1511] Processing CreateTablet for tablet a5bc21c3e9ab4003b697d645e55ef14d (DEFAULT_TABLE table=test-table [id=66b456ce820f433aa91cb9ab28982c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:58:57.670117 31946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5bc21c3e9ab4003b697d645e55ef14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:58:57.670692 32244 tablet_bootstrap.cc:654] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:57.672317 32244 tablet_bootstrap.cc:492] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592: No bootstrap required, opened a new log
I20260320 02:58:57.672487 32244 ts_tablet_manager.cc:1403] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260320 02:58:57.673067 32246 tablet_bootstrap.cc:492] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Bootstrap starting.
I20260320 02:58:57.673102 32244 raft_consensus.cc:359] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.673367 32244 raft_consensus.cc:385] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:57.673450 32244 raft_consensus.cc:740] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afdd0e581c244e6a0a63ca52c94a592, State: Initialized, Role: FOLLOWER
I20260320 02:58:57.673529 32243 tablet_bootstrap.cc:654] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:57.673702 32244 consensus_queue.cc:260] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.674211 32244 ts_tablet_manager.cc:1434] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260320 02:58:57.674727 32243 tablet_bootstrap.cc:492] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: No bootstrap required, opened a new log
I20260320 02:58:57.674825 32243 ts_tablet_manager.cc:1403] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260320 02:58:57.675088 32246 tablet_bootstrap.cc:654] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Neither blocks nor log segments found. Creating new log.
I20260320 02:58:57.675264 32243 raft_consensus.cc:359] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.675360 32243 raft_consensus.cc:385] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:57.675391 32243 raft_consensus.cc:740] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac75f983b52d436b89f01d48d87119bd, State: Initialized, Role: FOLLOWER
I20260320 02:58:57.675508 32243 consensus_queue.cc:260] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.675601 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.675226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:38505)
0320 02:58:57.675465 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.675523 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260320 02:58:57.675840 32243 ts_tablet_manager.cc:1434] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:58:57.676785 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.676490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:59421)
0320 02:58:57.676657 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.676715 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260320 02:58:57.678063 32246 tablet_bootstrap.cc:492] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: No bootstrap required, opened a new log
I20260320 02:58:57.678191 31876 heartbeater.cc:499] Master 127.23.132.124:42609 was elected leader, sending a full tablet report...
I20260320 02:58:57.678217 32246 ts_tablet_manager.cc:1403] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260320 02:58:57.678689 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.678057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:44955)
0320 02:58:57.678486 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.678569 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322}
I20260320 02:58:57.678659 32246 raft_consensus.cc:359] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.678763 32246 raft_consensus.cc:385] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:58:57.678799 32246 raft_consensus.cc:740] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf0ee39c16f4da6ad7a02c9c06fbaf3, State: Initialized, Role: FOLLOWER
I20260320 02:58:57.678965 32246 consensus_queue.cc:260] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.679190 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.678829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:40525)
0320 02:58:57.678996 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.679045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260320 02:58:57.679193 32148 heartbeater.cc:499] Master 127.23.132.124:42609 was elected leader, sending a full tablet report...
I20260320 02:58:57.679435 32246 ts_tablet_manager.cc:1434] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:58:57.680178 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.679948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:42139)
0320 02:58:57.680062 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.680111 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260320 02:58:57.681377 32012 heartbeater.cc:499] Master 127.23.132.124:42609 was elected leader, sending a full tablet report...
I20260320 02:58:57.681727 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.681517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:38775)
0320 02:58:57.681618 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.681663 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260320 02:58:57.836098 32249 raft_consensus.cc:493] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:58:57.836256 32249 raft_consensus.cc:515] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.836781 32249 leader_election.cc:290] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461), 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327)
I20260320 02:58:57.837525 31830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5bc21c3e9ab4003b697d645e55ef14d" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afdd0e581c244e6a0a63ca52c94a592" is_pre_election: true
I20260320 02:58:57.837570 31966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5bc21c3e9ab4003b697d645e55ef14d" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" is_pre_election: true
I20260320 02:58:57.837715 31830 raft_consensus.cc:2468] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 0.
I20260320 02:58:57.837759 31966 raft_consensus.cc:2468] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 0.
I20260320 02:58:57.838171 32036 leader_election.cc:304] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afdd0e581c244e6a0a63ca52c94a592, ac75f983b52d436b89f01d48d87119bd; no voters:
I20260320 02:58:57.838425 32249 raft_consensus.cc:2804] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:58:57.838490 32249 raft_consensus.cc:493] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:58:57.838526 32249 raft_consensus.cc:3060] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:57.839524 32249 raft_consensus.cc:515] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.839890 32249 leader_election.cc:290] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 election: Requested vote from peers 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461), 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327)
I20260320 02:58:57.840309 31966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5bc21c3e9ab4003b697d645e55ef14d" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3"
I20260320 02:58:57.840366 31830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5bc21c3e9ab4003b697d645e55ef14d" candidate_uuid: "ac75f983b52d436b89f01d48d87119bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afdd0e581c244e6a0a63ca52c94a592"
I20260320 02:58:57.840418 31966 raft_consensus.cc:3060] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:57.840482 31830 raft_consensus.cc:3060] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:58:57.841403 31966 raft_consensus.cc:2468] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 1.
I20260320 02:58:57.841696 32038 leader_election.cc:304] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf0ee39c16f4da6ad7a02c9c06fbaf3, ac75f983b52d436b89f01d48d87119bd; no voters:
I20260320 02:58:57.841879 32249 raft_consensus.cc:2804] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:58:57.842104 32249 raft_consensus.cc:697] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 1 LEADER]: Becoming Leader. State: Replica: ac75f983b52d436b89f01d48d87119bd, State: Running, Role: LEADER
I20260320 02:58:57.842258 32249 consensus_queue.cc:237] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } }
I20260320 02:58:57.842384 31830 raft_consensus.cc:2468] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac75f983b52d436b89f01d48d87119bd in term 1.
I20260320 02:58:57.843344 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.842848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:51799)
0320 02:58:57.843124 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.843224 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260320 02:58:57.844707 31681 catalog_manager.cc:5671] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac75f983b52d436b89f01d48d87119bd (127.23.132.67). New cstate: current_term: 1 leader_uuid: "ac75f983b52d436b89f01d48d87119bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac75f983b52d436b89f01d48d87119bd" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 36043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 } health_report { overall_health: UNKNOWN } } }
I20260320 02:58:57.861084 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.858170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.23.132.67:36043)
0320 02:58:57.858351 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0320 02:58:57.858354 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:58:57.858371 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:58:57.858420 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:58:57.858423 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:58:57.858472 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:58:57.858554 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:58:57.859147 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.859628 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:58:57.860351 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:58:57.860511 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:58:57.860560 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:58:57.860644 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:58:57.860705 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0320 02:58:57.860911 (+ 206us) server_negotiation.cc:300] Negotiation successful
0320 02:58:57.860962 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260320 02:58:57.864277 31830 raft_consensus.cc:1275] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Refusing update from remote peer ac75f983b52d436b89f01d48d87119bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:57.864703 31966 raft_consensus.cc:1275] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Refusing update from remote peer ac75f983b52d436b89f01d48d87119bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:58:57.864766 32252 consensus_queue.cc:1048] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afdd0e581c244e6a0a63ca52c94a592" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:57.865134 32252 consensus_queue.cc:1048] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf0ee39c16f4da6ad7a02c9c06fbaf3" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:58:57.867046 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.866619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:35333)
0320 02:58:57.866891 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.866963 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260320 02:58:57.867417 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.867128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:56203)
0320 02:58:57.867290 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.867344 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260320 02:58:57.871132 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.870732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:36709)
0320 02:58:57.870992 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.871047 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260320 02:58:57.872484 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.872040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.66:43019)
0320 02:58:57.872353 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.872408 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260320 02:58:57.877692 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.877294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.67:60977)
0320 02:58:57.877551 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.877609 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260320 02:58:57.878121 31682 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:57.878281 31682 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:58:57.879568 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.879300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:42639 (local address 127.23.132.65:39943)
0320 02:58:57.879436 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.879493 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260320 02:58:57.883467 31682 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 8221429cd61248029bfa740ab799e3f6: Sending DeleteTablet for 3 replicas of tablet a5bc21c3e9ab4003b697d645e55ef14d
I20260320 02:58:57.884254 32082 tablet_service.cc:1558] Processing DeleteTablet for tablet a5bc21c3e9ab4003b697d645e55ef14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:57 UTC) from {username='oryx', principal='oryx/127.23.132.124@KRBTEST.COM'} at 127.0.0.1:55522
I20260320 02:58:57.884338 31946 tablet_service.cc:1558] Processing DeleteTablet for tablet a5bc21c3e9ab4003b697d645e55ef14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:57 UTC) from {username='oryx', principal='oryx/127.23.132.124@KRBTEST.COM'} at 127.0.0.1:35476
I20260320 02:58:57.884616 31810 tablet_service.cc:1558] Processing DeleteTablet for tablet a5bc21c3e9ab4003b697d645e55ef14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:58:57 UTC) from {username='oryx', principal='oryx/127.23.132.124@KRBTEST.COM'} at 127.0.0.1:58364
I20260320 02:58:57.885043 32266 tablet_replica.cc:333] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: stopping tablet replica
I20260320 02:58:57.885061 32267 tablet_replica.cc:333] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592: stopping tablet replica
I20260320 02:58:57.885174 32267 raft_consensus.cc:2243] T a5bc21c3e9ab4003b697d645e55ef14d P 3afdd0e581c244e6a0a63ca52c94a592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:57.885228 32266 raft_consensus.cc:2243] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:58:57.885278 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 31747
I20260320 02:58:57.885425 32266 raft_consensus.cc:2272] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:57.886173 32265 tablet_replica.cc:333] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: stopping tablet replica
I20260320 02:58:57.886281 32265 raft_consensus.cc:2243] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:58:57.886468 32265 raft_consensus.cc:2272] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:58:57.887178 32265 ts_tablet_manager.cc:1916] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:57.887400 32266 ts_tablet_manager.cc:1916] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:58:57.889542 32266 ts_tablet_manager.cc:1929] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:57.889621 32266 log.cc:1199] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/a5bc21c3e9ab4003b697d645e55ef14d
I20260320 02:58:57.890293 32266 ts_tablet_manager.cc:1950] T a5bc21c3e9ab4003b697d645e55ef14d P 9cf0ee39c16f4da6ad7a02c9c06fbaf3: Deleting consensus metadata
I20260320 02:58:57.890370 32265 ts_tablet_manager.cc:1929] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:58:57.890496 32265 log.cc:1199] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/a5bc21c3e9ab4003b697d645e55ef14d
I20260320 02:58:57.890884 32265 ts_tablet_manager.cc:1950] T a5bc21c3e9ab4003b697d645e55ef14d P ac75f983b52d436b89f01d48d87119bd: Deleting consensus metadata
I20260320 02:58:57.891464 31670 catalog_manager.cc:5002] TS 9cf0ee39c16f4da6ad7a02c9c06fbaf3 (127.23.132.66:46461): tablet a5bc21c3e9ab4003b697d645e55ef14d (table test-table [id=66b456ce820f433aa91cb9ab28982c6b]) successfully deleted
I20260320 02:58:57.891690 31670 catalog_manager.cc:5002] TS ac75f983b52d436b89f01d48d87119bd (127.23.132.67:36043): tablet a5bc21c3e9ab4003b697d645e55ef14d (table test-table [id=66b456ce820f433aa91cb9ab28982c6b]) successfully deleted
W20260320 02:58:57.894600 31669 connection.cc:537] server connection from 127.23.132.65:50067 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260320 02:58:57.894848 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 31883
I20260320 02:58:57.897905 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:58:57.897129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:40327 (local address 127.0.0.1:58366)
0320 02:58:57.897440 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0320 02:58:57.897765 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.65:40327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260320 02:58:57.898178 31668 catalog_manager.cc:4729] TS 3afdd0e581c244e6a0a63ca52c94a592 (127.23.132.65:40327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5bc21c3e9ab4003b697d645e55ef14d: Network error: Client connection negotiation failed: client connection to 127.23.132.65:40327: connect: Connection refused (error 111)
I20260320 02:58:57.907217 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 32019
I20260320 02:58:57.915247 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 31358
I20260320 02:58:57.922247 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 31472
2026-03-20T02:58:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10102 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:58:58 dist-test-slave-v3hj krb5kdc[32270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:58:58 dist-test-slave-v3hj krb5kdc[32270](info): set up 2 sockets
Mar 20 02:58:58 dist-test-slave-v3hj krb5kdc[32270](info): commencing operation
krb5kdc: starting...
W20260320 02:59:00.115870 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.170s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:59:00 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:59:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:59:00.326968 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:45767
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:00.465561 32286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:00.465857 32286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:00.465976 32286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:00.471540 32286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:00.471638 32286 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260320 02:59:00.471665 32286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:00.471688 32286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:00.471707 32286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:00.471725 32286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:00.478138 32286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:45767
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:45767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:00.479339 32286 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:00.480161 32286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:00.488461 32291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:00.488464 32294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:00.488547 32286 server_base.cc:1061] running on GCE node
W20260320 02:59:00.488479 32292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:00.490041 32286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:00.492115 32286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:00.493319 32286 hybrid_clock.cc:648] HybridClock initialized: now 1773975540493245 us; error 96 us; skew 500 ppm
Mar 20 02:59:00 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975540, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:00.496903 32286 init.cc:377] Logged in from keytab as oryx/127.23.132.126@KRBTEST.COM (short username oryx)
I20260320 02:59:00.500600 32286 webserver.cc:492] Webserver started at http://127.23.132.126:37565/ using document root <none> and password file <none>
I20260320 02:59:00.501353 32286 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:00.501412 32286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:00.501588 32286 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:00.503682 32286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "54fa41b6692a406c8e42b873db738d1e"
format_stamp: "Formatted at 2026-03-20 02:59:00 on dist-test-slave-v3hj"
server_key: "09d08401ffdefba0676ccc900e129913"
server_key_iv: "83b9817d775aba8bdb655ded820fa7e4"
server_key_version: "encryptionkey@0"
I20260320 02:59:00.504132 32286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "54fa41b6692a406c8e42b873db738d1e"
format_stamp: "Formatted at 2026-03-20 02:59:00 on dist-test-slave-v3hj"
server_key: "09d08401ffdefba0676ccc900e129913"
server_key_iv: "83b9817d775aba8bdb655ded820fa7e4"
server_key_version: "encryptionkey@0"
I20260320 02:59:00.508409 32286 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:59:00.511507 32301 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:00.512919 32286 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 02:59:00.513067 32286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "54fa41b6692a406c8e42b873db738d1e"
format_stamp: "Formatted at 2026-03-20 02:59:00 on dist-test-slave-v3hj"
server_key: "09d08401ffdefba0676ccc900e129913"
server_key_iv: "83b9817d775aba8bdb655ded820fa7e4"
server_key_version: "encryptionkey@0"
I20260320 02:59:00.513159 32286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:00.530990 32286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:00.539265 32286 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:00.539443 32286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:00.555756 32286 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:45767
I20260320 02:59:00.556133 32363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:45767 every 8 connection(s)
I20260320 02:59:00.557397 32286 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:59:00.561444 32364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:00.566959 32364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Bootstrap starting.
I20260320 02:59:00.567533 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 32286
I20260320 02:59:00.567664 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:59:00.567901 24081 external_mini_cluster.cc:1468] Setting key 23faae2bd5f4d18a4d46e6ba2438b339
I20260320 02:59:00.571265 32364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:00.572906 32364 log.cc:826] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:00.576089 32364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: No bootstrap required, opened a new log
Mar 20 02:59:00 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:00.580753 32364 raft_consensus.cc:359] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fa41b6692a406c8e42b873db738d1e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45767 } }
I20260320 02:59:00.580991 32364 raft_consensus.cc:385] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:00.581043 32364 raft_consensus.cc:740] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fa41b6692a406c8e42b873db738d1e, State: Initialized, Role: FOLLOWER
I20260320 02:59:00.581775 32364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fa41b6692a406c8e42b873db738d1e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45767 } }
I20260320 02:59:00.583098 32364 raft_consensus.cc:399] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:59:00.583209 32364 raft_consensus.cc:493] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:59:00.583329 32364 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:00.584241 32364 raft_consensus.cc:515] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fa41b6692a406c8e42b873db738d1e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45767 } }
I20260320 02:59:00.584614 32364 leader_election.cc:304] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54fa41b6692a406c8e42b873db738d1e; no voters:
I20260320 02:59:00.584928 32364 leader_election.cc:290] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:59:00.585042 32369 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:00.585202 32369 raft_consensus.cc:697] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [term 1 LEADER]: Becoming Leader. State: Replica: 54fa41b6692a406c8e42b873db738d1e, State: Running, Role: LEADER
I20260320 02:59:00.585593 32369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fa41b6692a406c8e42b873db738d1e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45767 } }
I20260320 02:59:00.586486 32364 sys_catalog.cc:565] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:59:00.587550 32370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54fa41b6692a406c8e42b873db738d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fa41b6692a406c8e42b873db738d1e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45767 } } }
I20260320 02:59:00.587673 32370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:00.587738 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:00.570195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.23.132.126:45767)
0320 02:59:00.570900 (+ 705us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:00.570918 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:00.570960 (+ 42us) server_negotiation.cc:408] Connection header received
0320 02:59:00.571525 (+ 565us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:00.571541 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:00.571952 (+ 411us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:00.572320 (+ 368us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:00.573287 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:00.574118 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:00.575577 (+ 1459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:00.575892 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:00.578524 (+ 2632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:00.578554 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:00.578571 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:00.578629 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:00.582450 (+ 3821us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:00.583414 (+ 964us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:00.583420 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:00.583426 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:00.583523 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:00.584573 (+ 1050us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:00.584577 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:00.584579 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:00.585473 (+ 894us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:00.586496 (+ 1023us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:00.586720 (+ 224us) server_negotiation.cc:300] Negotiation successful
0320 02:59:00.586967 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":326,"threads_started":1}
I20260320 02:59:00.589509 32371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54fa41b6692a406c8e42b873db738d1e. Latest consensus state: current_term: 1 leader_uuid: "54fa41b6692a406c8e42b873db738d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54fa41b6692a406c8e42b873db738d1e" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45767 } } }
I20260320 02:59:00.589690 32371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e [sys.catalog]: This master's current role is: LEADER
W20260320 02:59:00.593639 32385 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:59:00.593828 32385 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:59:00.593927 32378 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:59:00.595180 32378 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:59:00.606740 32378 catalog_manager.cc:1357] Generated new cluster ID: 9e96110e418d414dafa387333c9a32df
I20260320 02:59:00.606904 32378 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:59:00.632910 32378 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:59:00.634459 32378 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:59:00.651539 32378 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Generated new TSK 0
I20260320 02:59:00.653942 32378 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:59:00.762810 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:45767
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:00.927158 32392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:00.927400 32392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:00.927462 32392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:00.932648 32392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:00.932786 32392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:00.932978 32392 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:59:00.933324 32392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:59:00.941355 32392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:00.942849 32392 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:00.943897 32392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:00.952911 32397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:00.953094 32400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:00.953925 32398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:00.954277 32392 server_base.cc:1061] running on GCE node
I20260320 02:59:00.954808 32392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:00.955863 32392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:00.957944 32392 hybrid_clock.cc:648] HybridClock initialized: now 1773975540957819 us; error 127 us; skew 500 ppm
Mar 20 02:59:00 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975540, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:00.962208 32392 init.cc:377] Logged in from keytab as oryx/127.23.132.65@KRBTEST.COM (short username oryx)
I20260320 02:59:00.963441 32392 webserver.cc:492] Webserver started at http://127.23.132.65:34843/ using document root <none> and password file <none>
I20260320 02:59:00.964007 32392 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:00.964075 32392 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:00.964251 32392 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:00.965997 32392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "0869b28899a543bf816956fd5667093d"
format_stamp: "Formatted at 2026-03-20 02:59:00 on dist-test-slave-v3hj"
server_key: "cddd8d15e546ac23860d7634159c5f60"
server_key_iv: "29688e41825a525676e980f587a2c601"
server_key_version: "encryptionkey@0"
I20260320 02:59:00.966428 32392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "0869b28899a543bf816956fd5667093d"
format_stamp: "Formatted at 2026-03-20 02:59:00 on dist-test-slave-v3hj"
server_key: "cddd8d15e546ac23860d7634159c5f60"
server_key_iv: "29688e41825a525676e980f587a2c601"
server_key_version: "encryptionkey@0"
I20260320 02:59:00.971060 32392 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:00.974906 32407 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:00.976503 32392 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:00.976634 32392 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "0869b28899a543bf816956fd5667093d"
format_stamp: "Formatted at 2026-03-20 02:59:00 on dist-test-slave-v3hj"
server_key: "cddd8d15e546ac23860d7634159c5f60"
server_key_iv: "29688e41825a525676e980f587a2c601"
server_key_version: "encryptionkey@0"
I20260320 02:59:00.977190 32392 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:00.994817 32392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:00.999455 32392 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:00.999668 32392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:01.001225 32392 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:01.001286 32392 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:01.001341 32392 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:01.001365 32392 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975540, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:01.014576 32522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:41945 every 8 connection(s)
I20260320 02:59:01.014758 32392 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:41945
I20260320 02:59:01.016326 32392 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:59:01.022049 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.003267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:47695 (local address 127.23.132.126:45767)
0320 02:59:01.003393 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.003396 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.004286 (+ 890us) server_negotiation.cc:408] Connection header received
0320 02:59:01.005666 (+ 1380us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.005672 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.005740 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.005867 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.008125 (+ 2258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.008915 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.010574 (+ 1659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.010772 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.015397 (+ 4625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.015425 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.015429 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.015464 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.018238 (+ 2774us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.019609 (+ 1371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.019615 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.019617 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.019704 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.020260 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.020265 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.020267 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.020471 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.020584 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.021687 (+ 1103us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.021903 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260320 02:59:01.023118 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.003608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.23.132.65:47695)
0320 02:59:01.004133 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.004180 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.005411 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.006191 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.006204 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.007022 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:01.007893 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.007913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.009089 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.009092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.010402 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.010414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.010810 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.012360 (+ 1550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:01.012404 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:01.015210 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:01.019023 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.019120 (+ 97us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.019132 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.019433 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.019994 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.019998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.019999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.020105 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.020694 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:01.020715 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:01.021324 (+ 609us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.021583 (+ 259us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.022402 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260320 02:59:01.024484 32523 heartbeater.cc:344] Connected to a master server at 127.23.132.126:45767
I20260320 02:59:01.024655 32523 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:01.025728 32523 heartbeater.cc:507] Master 127.23.132.126:45767 requested a full tablet report, sending...
I20260320 02:59:01.027158 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 32392
I20260320 02:59:01.027294 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:59:01.027583 24081 external_mini_cluster.cc:1468] Setting key e7f7a73fcf6c8609ac275c1e3fb6754a
I20260320 02:59:01.028668 32317 ts_manager.cc:194] Registered new tserver with Master: 0869b28899a543bf816956fd5667093d (127.23.132.65:41945)
I20260320 02:59:01.030880 32317 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:47695
WARNING: no policy specified for oryx/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:59:01.045738 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.028695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:41647 (local address 127.23.132.126:45767)
0320 02:59:01.028867 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.028871 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.029965 (+ 1094us) server_negotiation.cc:408] Connection header received
0320 02:59:01.030039 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.030043 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.030089 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.030158 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.031255 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.032917 (+ 1662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.033580 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.033869 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.035710 (+ 1841us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.035750 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.035754 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.035805 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.040345 (+ 4540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.041510 (+ 1165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.041514 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.041517 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.041582 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.043635 (+ 2053us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.043640 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.043642 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.044103 (+ 461us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.044221 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.045192 (+ 971us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.045343 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260320 02:59:01.046195 32529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.028506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.23.132.65:41647)
0320 02:59:01.029785 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.029838 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.029946 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.030190 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.030193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.030671 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:01.031136 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.031146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.033062 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.033066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.033472 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.033480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.033588 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.034581 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:01.034595 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:01.035337 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:01.040813 (+ 5476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.040818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.040820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.041071 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.042139 (+ 1068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.042146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.042148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.042712 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.044285 (+ 1573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:01.044288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:01.044717 (+ 429us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.045123 (+ 406us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.045855 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:59:01.119192 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:45767
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:01.259387 32534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:01.259718 32534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:01.259811 32534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:01.264102 32534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:01.264199 32534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:01.264282 32534 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:59:01.264350 32534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:59:01.269179 32534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:01.271140 32534 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:01.272508 32534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:01.280336 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:01.280336 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:01.280339 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:01.280871 32534 server_base.cc:1061] running on GCE node
I20260320 02:59:01.281636 32534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:01.282251 32534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:01.283478 32534 hybrid_clock.cc:648] HybridClock initialized: now 1773975541283412 us; error 65 us; skew 500 ppm
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975541, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:01.287039 32534 init.cc:377] Logged in from keytab as oryx/127.23.132.66@KRBTEST.COM (short username oryx)
I20260320 02:59:01.288373 32534 webserver.cc:492] Webserver started at http://127.23.132.66:39525/ using document root <none> and password file <none>
I20260320 02:59:01.288945 32534 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:01.289011 32534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:01.289209 32534 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:01.291139 32534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "a32aeba715c645cb97a174f435a187e5"
format_stamp: "Formatted at 2026-03-20 02:59:01 on dist-test-slave-v3hj"
server_key: "86ef34424d63b68eb48060ee5c167f69"
server_key_iv: "e9ba6a296a521b4ef3345dfe96d00935"
server_key_version: "encryptionkey@0"
I20260320 02:59:01.291604 32534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "a32aeba715c645cb97a174f435a187e5"
format_stamp: "Formatted at 2026-03-20 02:59:01 on dist-test-slave-v3hj"
server_key: "86ef34424d63b68eb48060ee5c167f69"
server_key_iv: "e9ba6a296a521b4ef3345dfe96d00935"
server_key_version: "encryptionkey@0"
I20260320 02:59:01.295459 32534 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:01.298055 32549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:01.299703 32534 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260320 02:59:01.299943 32534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "a32aeba715c645cb97a174f435a187e5"
format_stamp: "Formatted at 2026-03-20 02:59:01 on dist-test-slave-v3hj"
server_key: "86ef34424d63b68eb48060ee5c167f69"
server_key_iv: "e9ba6a296a521b4ef3345dfe96d00935"
server_key_version: "encryptionkey@0"
I20260320 02:59:01.300079 32534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:01.316491 32534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:01.322306 32534 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:01.322484 32534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:01.324364 32534 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:01.324435 32534 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:01.324479 32534 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:01.324492 32534 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975541, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:01.340128 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.325301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:57009 (local address 127.23.132.126:45767)
0320 02:59:01.325539 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.325545 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.326785 (+ 1240us) server_negotiation.cc:408] Connection header received
0320 02:59:01.328218 (+ 1433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.328224 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.328643 (+ 419us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.328919 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.330930 (+ 2011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.331439 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.332110 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.332281 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.335760 (+ 3479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.335784 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.335788 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.335818 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.337442 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.337971 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.338018 (+ 47us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.338021 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.338080 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.338527 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.338531 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.338533 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.338705 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.338797 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.339699 (+ 902us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.339903 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260320 02:59:01.341198 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.325836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.23.132.66:57009)
0320 02:59:01.326569 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.326632 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.327669 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.329274 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.329284 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.330128 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:01.330729 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.330743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.331563 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.331566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.331991 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.331998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.332201 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.332892 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:01.332929 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:01.335543 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:01.337568 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.337575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.337588 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.337862 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.338288 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.338298 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.338299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.338404 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.338822 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:01.338829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:01.339264 (+ 435us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.339561 (+ 297us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.339955 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260320 02:59:01.342430 32534 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:36383
I20260320 02:59:01.342459 32664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:36383 every 8 connection(s)
I20260320 02:59:01.343719 32534 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:59:01.345885 32669 heartbeater.cc:344] Connected to a master server at 127.23.132.126:45767
I20260320 02:59:01.346122 32669 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:01.346654 32669 heartbeater.cc:507] Master 127.23.132.126:45767 requested a full tablet report, sending...
I20260320 02:59:01.347795 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 32534
I20260320 02:59:01.347900 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:59:01.348142 24081 external_mini_cluster.cc:1468] Setting key acc51e6867499ca49eaa4ac4763c5543
I20260320 02:59:01.348146 32317 ts_manager.cc:194] Registered new tserver with Master: a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:01.349160 32317 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:57009
I20260320 02:59:01.356906 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.345588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:45669 (local address 127.23.132.126:45767)
0320 02:59:01.345900 (+ 312us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.345904 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.346072 (+ 168us) server_negotiation.cc:408] Connection header received
0320 02:59:01.346224 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.346228 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.346280 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.346436 (+ 156us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.347261 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.348420 (+ 1159us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.349400 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.349574 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.351803 (+ 2229us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.351832 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.351836 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.351870 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.353973 (+ 2103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.354711 (+ 738us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.354715 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.354716 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.354767 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.355239 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.355244 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.355245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.355655 (+ 410us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.356051 (+ 396us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.356441 (+ 390us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.356562 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260320 02:59:01.356916 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.345550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.23.132.66:45669)
0320 02:59:01.345992 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.346009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.346122 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.346472 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.346475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.346776 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:01.347060 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.347069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.348537 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.348541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.349287 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.349297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.349401 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.351112 (+ 1711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:01.351133 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:01.351603 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:01.354213 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.354218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.354220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.354465 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.354885 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.354888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.354889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.354970 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.356070 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:01.356074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:01.356170 (+ 96us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.356267 (+ 97us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.356533 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
WARNING: no policy specified for oryx/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab.
I20260320 02:59:01.429774 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:45767
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:01.575102 32676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:01.575348 32676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:01.575410 32676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:01.579082 32676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:01.579376 32676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:01.579466 32676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260320 02:59:01.579532 32676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:59:01.584960 32676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/krb5kdc/oryx.keytab
--principal=oryx/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.132.126:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:01.586171 32676 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:01.587013 32676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:01.596318 32681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:01.596333 32682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:01.596331 32684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:01.599156 32676 server_base.cc:1061] running on GCE node
I20260320 02:59:01.599702 32676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:01.600411 32676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:01.601616 32676 hybrid_clock.cc:648] HybridClock initialized: now 1773975541601602 us; error 41 us; skew 500 ppm
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975541, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:01.605526 32676 init.cc:377] Logged in from keytab as oryx/127.23.132.67@KRBTEST.COM (short username oryx)
I20260320 02:59:01.607703 32676 webserver.cc:492] Webserver started at http://127.23.132.67:38901/ using document root <none> and password file <none>
I20260320 02:59:01.608467 32676 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:01.608536 32676 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:01.608764 32676 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:01.611068 32676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7"
format_stamp: "Formatted at 2026-03-20 02:59:01 on dist-test-slave-v3hj"
server_key: "201c1a458c07bcc90ef653783e4f1e7e"
server_key_iv: "5163d4ce57900d0d48e33f8bb8b154cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:01.611652 32676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7"
format_stamp: "Formatted at 2026-03-20 02:59:01 on dist-test-slave-v3hj"
server_key: "201c1a458c07bcc90ef653783e4f1e7e"
server_key_iv: "5163d4ce57900d0d48e33f8bb8b154cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:01.615674 32676 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:01.618458 32691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:01.621187 32676 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:01.621316 32676 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7"
format_stamp: "Formatted at 2026-03-20 02:59:01 on dist-test-slave-v3hj"
server_key: "201c1a458c07bcc90ef653783e4f1e7e"
server_key_iv: "5163d4ce57900d0d48e33f8bb8b154cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:01.621423 32676 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:01.636847 32676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:01.641471 32676 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:01.641664 32676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:01.644004 32676 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:01.644299 32676 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:01.644421 32676 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:01.644464 32676 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975541, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:01.659777 32676 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:38205
I20260320 02:59:01.660447 338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:38205 every 8 connection(s)
I20260320 02:59:01.660943 32676 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:59:01.661854 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.644781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.23.132.67:43651)
0320 02:59:01.645354 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.645388 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.646303 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.646851 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.646860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.647386 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:01.647963 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.647977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.649206 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.649210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.651015 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.651047 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.651296 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.652817 (+ 1521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:01.652882 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:01.655363 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:01.658626 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.658633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.658648 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.659010 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.659443 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.659448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.659450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.659576 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.660182 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:01.660189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:01.660467 (+ 278us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.660695 (+ 228us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.660997 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":128,"threads_started":1}
I20260320 02:59:01.662635 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.644280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:43651 (local address 127.23.132.126:45767)
0320 02:59:01.644441 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.644445 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.645507 (+ 1062us) server_negotiation.cc:408] Connection header received
0320 02:59:01.646518 (+ 1011us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.646522 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.646585 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.646677 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.648126 (+ 1449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.648826 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.651287 (+ 2461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.651765 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.655609 (+ 3844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.655638 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.655641 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.655679 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.658443 (+ 2764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.659133 (+ 690us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.659137 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.659139 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.659191 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.659784 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.659787 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.659789 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.660032 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.660128 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.661914 (+ 1786us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.662485 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260320 02:59:01.663929 339 heartbeater.cc:344] Connected to a master server at 127.23.132.126:45767
I20260320 02:59:01.664320 339 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:01.665020 339 heartbeater.cc:507] Master 127.23.132.126:45767 requested a full tablet report, sending...
I20260320 02:59:01.666144 32318 ts_manager.cc:194] Registered new tserver with Master: a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205)
I20260320 02:59:01.667800 32318 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:43651
I20260320 02:59:01.669443 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 32676
I20260320 02:59:01.669560 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:59:01.669835 24081 external_mini_cluster.cc:1468] Setting key 0a36306fa62d96e324dc795214653454
I20260320 02:59:01.675150 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:59:01.680330 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.670468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:54255 (local address 127.23.132.126:45767)
0320 02:59:01.670713 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.670718 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.670926 (+ 208us) server_negotiation.cc:408] Connection header received
0320 02:59:01.670973 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.670976 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.671126 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.671224 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.672085 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.672798 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.674694 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.674882 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.675207 (+ 325us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.675226 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.675229 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.675256 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.677664 (+ 2408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.678922 (+ 1258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.678926 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.678928 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.678970 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.679438 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.679442 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.679444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.679598 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.679801 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.680065 (+ 264us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.680171 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260320 02:59:01.680346 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.668601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.23.132.67:54255)
0320 02:59:01.669039 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.669054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.669161 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.671350 (+ 2189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.671353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.671571 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:01.671934 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.671945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.673010 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.673014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.673642 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.673665 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.673998 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.674689 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:01.674706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:01.675087 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:01.678622 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.678628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.678629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.678814 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.679292 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:01.679296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:01.679298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:01.679354 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:01.679803 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:01.679807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:01.679907 (+ 100us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.679999 (+ 92us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.680156 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":145,"threads_started":1}
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:01.704548 32367 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:01.697581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.23.132.126:45767)
0320 02:59:01.698084 (+ 503us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.698089 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.698108 (+ 19us) server_negotiation.cc:408] Connection header received
0320 02:59:01.698151 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.698154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.698201 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.698284 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.699490 (+ 1206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.700181 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.701095 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.701372 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.704430 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56706: BlockingRecv error: recv got EOF from 127.0.0.1:56706 (error 108)
Metrics: {"server-negotiator.queue_time_us":331}
Mar 20 02:59:01 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:01.721385 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.707992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.23.132.126:45767)
0320 02:59:01.708456 (+ 464us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.708460 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.708476 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:01.708517 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.708520 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.708564 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.709516 (+ 952us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:01.711004 (+ 1488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.711833 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.712944 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.713120 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.716669 (+ 3549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:01.716706 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:01.716708 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:01.716757 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:01.718977 (+ 2220us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.719516 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.719520 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.719521 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.719571 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:01.719780 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:01.719783 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:01.719785 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:01.720201 (+ 416us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:01.720409 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.720844 (+ 435us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.721050 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260320 02:59:01.726254 32318 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:59:01.729365 32318 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:59:01.747113 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.742116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:41945 (local address 127.0.0.1:51976)
0320 02:59:01.742934 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.742945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.743024 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.743391 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.743396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.743416 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:01.743961 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.743988 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.745691 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.745699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.746516 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.746527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.746661 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.746743 (+ 82us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.746836 (+ 93us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.746939 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":79,"threads_started":1}
I20260320 02:59:01.747362 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.741635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:36383 (local address 127.0.0.1:40554)
0320 02:59:01.742570 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.742608 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.742760 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.743405 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.743408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.743426 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:01.743739 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.743747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.745552 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.745556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.746567 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.746578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.746715 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.746768 (+ 53us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.746832 (+ 64us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.746940 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":141,"threads_started":1}
I20260320 02:59:01.747748 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.741869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:38205 (local address 127.0.0.1:49078)
0320 02:59:01.742920 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.742933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.743057 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.743504 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.743507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.743520 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:01.743746 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.743751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.745594 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.745597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.746570 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.746578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.747487 (+ 909us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.747503 (+ 16us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.747572 (+ 69us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.747633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":116,"threads_started":1}
I20260320 02:59:01.748181 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.742449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.23.132.67:38205)
0320 02:59:01.742805 (+ 356us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.742811 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.743003 (+ 192us) server_negotiation.cc:408] Connection header received
0320 02:59:01.743215 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.743219 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.743339 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.743499 (+ 160us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:01.744057 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.745467 (+ 1410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.746759 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.747300 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.747423 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.747949 (+ 526us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.748042 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260320 02:59:01.748188 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.741810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.23.132.66:36383)
0320 02:59:01.742198 (+ 388us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.742205 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.742778 (+ 573us) server_negotiation.cc:408] Connection header received
0320 02:59:01.742899 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.742904 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.743092 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.743221 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:01.743885 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.745413 (+ 1528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.747033 (+ 1620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.747803 (+ 770us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.747903 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.747957 (+ 54us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.748042 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":111,"threads_started":1}
I20260320 02:59:01.748191 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.742484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.23.132.65:41945)
0320 02:59:01.743025 (+ 541us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.743030 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.743049 (+ 19us) server_negotiation.cc:408] Connection header received
0320 02:59:01.743117 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.743122 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.743250 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.743364 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:01.744185 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.745413 (+ 1228us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.746684 (+ 1271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.747679 (+ 995us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.747781 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.747951 (+ 170us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.748061 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":80,"threads_started":1}
I20260320 02:59:01.752094 32599 tablet_service.cc:1511] Processing CreateTablet for tablet c46d3e255fbe4211a3560a5edd7cd4cc (DEFAULT_TABLE table=test-table [id=14a2d2e4d5d44d08b72f7e4a152c14da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:01.752072 32454 tablet_service.cc:1511] Processing CreateTablet for tablet c46d3e255fbe4211a3560a5edd7cd4cc (DEFAULT_TABLE table=test-table [id=14a2d2e4d5d44d08b72f7e4a152c14da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:01.751942 32741 tablet_service.cc:1511] Processing CreateTablet for tablet c46d3e255fbe4211a3560a5edd7cd4cc (DEFAULT_TABLE table=test-table [id=14a2d2e4d5d44d08b72f7e4a152c14da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:01.753181 32454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46d3e255fbe4211a3560a5edd7cd4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:01.753181 32599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46d3e255fbe4211a3560a5edd7cd4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:01.753602 32741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46d3e255fbe4211a3560a5edd7cd4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:01.761209 368 tablet_bootstrap.cc:492] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Bootstrap starting.
I20260320 02:59:01.761898 369 tablet_bootstrap.cc:492] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Bootstrap starting.
I20260320 02:59:01.764433 370 tablet_bootstrap.cc:492] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Bootstrap starting.
I20260320 02:59:01.765328 368 tablet_bootstrap.cc:654] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:01.765376 369 tablet_bootstrap.cc:654] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:01.766464 368 log.cc:826] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:01.767006 369 log.cc:826] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:01.767307 370 tablet_bootstrap.cc:654] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:01.768647 370 log.cc:826] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:01.769611 368 tablet_bootstrap.cc:492] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: No bootstrap required, opened a new log
I20260320 02:59:01.769738 369 tablet_bootstrap.cc:492] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: No bootstrap required, opened a new log
I20260320 02:59:01.769999 369 ts_tablet_manager.cc:1403] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260320 02:59:01.770581 368 ts_tablet_manager.cc:1403] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260320 02:59:01.772784 369 raft_consensus.cc:359] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.772989 369 raft_consensus.cc:385] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:01.773064 369 raft_consensus.cc:740] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32aeba715c645cb97a174f435a187e5, State: Initialized, Role: FOLLOWER
I20260320 02:59:01.773510 369 consensus_queue.cc:260] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.773896 370 tablet_bootstrap.cc:492] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: No bootstrap required, opened a new log
I20260320 02:59:01.774160 370 ts_tablet_manager.cc:1403] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260320 02:59:01.774471 32669 heartbeater.cc:499] Master 127.23.132.126:45767 was elected leader, sending a full tablet report...
I20260320 02:59:01.774734 369 ts_tablet_manager.cc:1434] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260320 02:59:01.775524 368 raft_consensus.cc:359] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.775830 368 raft_consensus.cc:385] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:01.775929 368 raft_consensus.cc:740] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0869b28899a543bf816956fd5667093d, State: Initialized, Role: FOLLOWER
I20260320 02:59:01.776451 368 consensus_queue.cc:260] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.777377 368 ts_tablet_manager.cc:1434] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260320 02:59:01.777328 370 raft_consensus.cc:359] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.777530 370 raft_consensus.cc:385] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:01.777626 370 raft_consensus.cc:740] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Initialized, Role: FOLLOWER
I20260320 02:59:01.777930 32523 heartbeater.cc:499] Master 127.23.132.126:45767 was elected leader, sending a full tablet report...
I20260320 02:59:01.778227 370 consensus_queue.cc:260] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.778946 339 heartbeater.cc:499] Master 127.23.132.126:45767 was elected leader, sending a full tablet report...
I20260320 02:59:01.779103 370 ts_tablet_manager.cc:1434] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260320 02:59:01.818863 376 raft_consensus.cc:493] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:59:01.819356 376 raft_consensus.cc:515] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.821307 376 leader_election.cc:290] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383), 0869b28899a543bf816956fd5667093d (127.23.132.65:41945)
I20260320 02:59:01.824883 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.821754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:41945 (local address 127.23.132.67:43175)
0320 02:59:01.822124 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.822135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.822389 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.822927 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.822930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.822948 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:01.823221 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.823227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.823899 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.823902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.824475 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.824482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.824588 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.824612 (+ 24us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.824663 (+ 51us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.824722 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":144,"threads_started":1}
I20260320 02:59:01.825070 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.821454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:36383 (local address 127.23.132.67:53435)
0320 02:59:01.821584 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:01.821600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:01.821757 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:01.822191 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:01.822194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:01.822210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:01.822523 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.822530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.823408 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.823607 (+ 199us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:01.824201 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:01.824211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.824808 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.824827 (+ 19us) client_negotiation.cc:769] Sending connection context
0320 02:59:01.824890 (+ 63us) client_negotiation.cc:240] Negotiation successful
0320 02:59:01.824950 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":163712}
I20260320 02:59:01.825304 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.821813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:43175 (local address 127.23.132.65:41945)
0320 02:59:01.822000 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.822004 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.822212 (+ 208us) server_negotiation.cc:408] Connection header received
0320 02:59:01.822648 (+ 436us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.822652 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.822697 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.822903 (+ 206us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:01.823332 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.823800 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.824586 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.825045 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.825086 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.825135 (+ 49us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.825195 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260320 02:59:01.825520 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.821784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:53435 (local address 127.23.132.66:36383)
0320 02:59:01.821981 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.821985 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.822000 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:59:01.822051 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.822053 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.822092 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.822159 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:01.822766 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.823288 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.824789 (+ 1501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.825265 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.825302 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.825353 (+ 51us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.825413 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260320 02:59:01.826048 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c46d3e255fbe4211a3560a5edd7cd4cc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5" is_pre_election: true
I20260320 02:59:01.826254 32477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c46d3e255fbe4211a3560a5edd7cd4cc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d" is_pre_election: true
I20260320 02:59:01.826370 32619 raft_consensus.cc:2468] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:01.826468 32477 raft_consensus.cc:2468] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:01.827001 32695 leader_election.cc:304] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a0c8fe59e39f4fecb14c1a0fd79373b7, a32aeba715c645cb97a174f435a187e5; no voters:
I20260320 02:59:01.827363 376 raft_consensus.cc:2804] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:59:01.827556 376 raft_consensus.cc:493] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:59:01.827612 376 raft_consensus.cc:3060] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:01.829005 376 raft_consensus.cc:515] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.829514 376 leader_election.cc:290] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Requested vote from peers a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383), 0869b28899a543bf816956fd5667093d (127.23.132.65:41945)
I20260320 02:59:01.830094 32477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c46d3e255fbe4211a3560a5edd7cd4cc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d"
I20260320 02:59:01.830096 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c46d3e255fbe4211a3560a5edd7cd4cc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5"
I20260320 02:59:01.830217 32619 raft_consensus.cc:3060] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:01.830224 32477 raft_consensus.cc:3060] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:01.831234 32619 raft_consensus.cc:2468] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:01.831565 32693 leader_election.cc:304] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c8fe59e39f4fecb14c1a0fd79373b7, a32aeba715c645cb97a174f435a187e5; no voters:
I20260320 02:59:01.832059 376 raft_consensus.cc:2804] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:01.832306 376 raft_consensus.cc:697] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Becoming Leader. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Running, Role: LEADER
I20260320 02:59:01.833148 376 consensus_queue.cc:237] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } }
I20260320 02:59:01.833747 32477 raft_consensus.cc:2468] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:01.838675 32318 catalog_manager.cc:5671] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } health_report { overall_health: UNKNOWN } } }
W20260320 02:59:01.845309 32670 tablet.cc:2379] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:59:01.861163 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:01.856465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.23.132.67:38205)
0320 02:59:01.856594 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:01.856598 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:01.856612 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:59:01.857122 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:01.857126 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:01.857185 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:01.857276 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:01.857820 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.858445 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:01.859458 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:01.859663 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:01.859725 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:01.860394 (+ 669us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:01.860751 (+ 357us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:01.860986 (+ 235us) server_negotiation.cc:300] Negotiation successful
0320 02:59:01.861042 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260320 02:59:01.868789 32619 raft_consensus.cc:1275] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:01.868777 32477 raft_consensus.cc:1275] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:01.869745 378 consensus_queue.cc:1048] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:01.869998 376 consensus_queue.cc:1048] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:01.883795 383 mvcc.cc:204] Tried to move back new op lower bound from 7266203819484139520 to 7266203819356758016. Current Snapshot: MvccSnapshot[applied={T|T < 7266203819484139520}]
I20260320 02:59:01.887437 382 mvcc.cc:204] Tried to move back new op lower bound from 7266203819484139520 to 7266203819356758016. Current Snapshot: MvccSnapshot[applied={T|T < 7266203819484139520 or (T in {7266203819484139520})}]
I20260320 02:59:01.897524 32316 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:01.897716 32316 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:01.901153 32316 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Sending DeleteTablet for 3 replicas of tablet c46d3e255fbe4211a3560a5edd7cd4cc
I20260320 02:59:01.902222 32454 tablet_service.cc:1558] Processing DeleteTablet for tablet c46d3e255fbe4211a3560a5edd7cd4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:01 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:51976
I20260320 02:59:01.902318 32741 tablet_service.cc:1558] Processing DeleteTablet for tablet c46d3e255fbe4211a3560a5edd7cd4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:01 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:49078
I20260320 02:59:01.902415 32599 tablet_service.cc:1558] Processing DeleteTablet for tablet c46d3e255fbe4211a3560a5edd7cd4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:01 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:40554
I20260320 02:59:01.903395 393 tablet_replica.cc:333] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: stopping tablet replica
I20260320 02:59:01.903551 392 tablet_replica.cc:333] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: stopping tablet replica
I20260320 02:59:01.903743 392 raft_consensus.cc:2243] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:59:01.903782 393 raft_consensus.cc:2243] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:01.904130 393 raft_consensus.cc:2272] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:01.904182 392 raft_consensus.cc:2272] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:01.904541 32316 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:59:01.905052 32316 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:59:01.905684 392 ts_tablet_manager.cc:1916] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:01.905684 393 ts_tablet_manager.cc:1916] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:01.907267 391 tablet_replica.cc:333] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: stopping tablet replica
I20260320 02:59:01.907632 391 raft_consensus.cc:2243] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:01.908051 391 raft_consensus.cc:2272] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:01.910432 393 ts_tablet_manager.cc:1929] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:01.910538 393 log.cc:1199] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/c46d3e255fbe4211a3560a5edd7cd4cc
I20260320 02:59:01.910852 393 ts_tablet_manager.cc:1950] T c46d3e255fbe4211a3560a5edd7cd4cc P 0869b28899a543bf816956fd5667093d: Deleting consensus metadata
I20260320 02:59:01.913045 32741 tablet_service.cc:1511] Processing CreateTablet for tablet 3887a4fce9164d289e61d4243b6b87ac (DEFAULT_TABLE table=test-table [id=ce46c7a442d6437dbd84fa37f5e3c85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:01.913045 32454 tablet_service.cc:1511] Processing CreateTablet for tablet 3887a4fce9164d289e61d4243b6b87ac (DEFAULT_TABLE table=test-table [id=ce46c7a442d6437dbd84fa37f5e3c85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:01.913467 32454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887a4fce9164d289e61d4243b6b87ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:01.913398 32599 tablet_service.cc:1511] Processing CreateTablet for tablet 3887a4fce9164d289e61d4243b6b87ac (DEFAULT_TABLE table=test-table [id=ce46c7a442d6437dbd84fa37f5e3c85f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:01.914324 32741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887a4fce9164d289e61d4243b6b87ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:01.914429 32599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887a4fce9164d289e61d4243b6b87ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:01.917408 368 tablet_bootstrap.cc:492] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Bootstrap starting.
I20260320 02:59:01.919121 32305 catalog_manager.cc:5002] TS 0869b28899a543bf816956fd5667093d (127.23.132.65:41945): tablet c46d3e255fbe4211a3560a5edd7cd4cc (table test-table [id=14a2d2e4d5d44d08b72f7e4a152c14da]) successfully deleted
I20260320 02:59:01.919435 368 tablet_bootstrap.cc:654] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:01.919467 392 ts_tablet_manager.cc:1929] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:01.919552 392 log.cc:1199] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/c46d3e255fbe4211a3560a5edd7cd4cc
I20260320 02:59:01.919900 392 ts_tablet_manager.cc:1950] T c46d3e255fbe4211a3560a5edd7cd4cc P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting consensus metadata
I20260320 02:59:01.920007 391 ts_tablet_manager.cc:1916] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:01.920243 370 tablet_bootstrap.cc:492] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Bootstrap starting.
I20260320 02:59:01.921329 370 tablet_bootstrap.cc:654] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:01.921629 32304 catalog_manager.cc:5002] TS a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205): tablet c46d3e255fbe4211a3560a5edd7cd4cc (table test-table [id=14a2d2e4d5d44d08b72f7e4a152c14da]) successfully deleted
I20260320 02:59:01.923020 391 ts_tablet_manager.cc:1929] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:01.923105 391 log.cc:1199] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/c46d3e255fbe4211a3560a5edd7cd4cc
I20260320 02:59:01.923312 391 ts_tablet_manager.cc:1950] T c46d3e255fbe4211a3560a5edd7cd4cc P a32aeba715c645cb97a174f435a187e5: Deleting consensus metadata
I20260320 02:59:01.925549 32303 catalog_manager.cc:5002] TS a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383): tablet c46d3e255fbe4211a3560a5edd7cd4cc (table test-table [id=14a2d2e4d5d44d08b72f7e4a152c14da]) successfully deleted
I20260320 02:59:01.926327 368 tablet_bootstrap.cc:492] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: No bootstrap required, opened a new log
I20260320 02:59:01.926445 368 ts_tablet_manager.cc:1403] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260320 02:59:01.927404 368 raft_consensus.cc:359] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:01.927618 368 raft_consensus.cc:385] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:01.927696 368 raft_consensus.cc:740] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0869b28899a543bf816956fd5667093d, State: Initialized, Role: FOLLOWER
I20260320 02:59:01.927834 368 consensus_queue.cc:260] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:01.929210 368 ts_tablet_manager.cc:1434] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260320 02:59:01.931722 370 tablet_bootstrap.cc:492] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: No bootstrap required, opened a new log
I20260320 02:59:01.931773 369 tablet_bootstrap.cc:492] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Bootstrap starting.
I20260320 02:59:01.931844 370 ts_tablet_manager.cc:1403] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260320 02:59:01.932334 370 raft_consensus.cc:359] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:01.932446 370 raft_consensus.cc:385] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:01.932472 370 raft_consensus.cc:740] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Initialized, Role: FOLLOWER
I20260320 02:59:01.932690 370 consensus_queue.cc:260] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:01.932854 369 tablet_bootstrap.cc:654] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:01.932996 370 ts_tablet_manager.cc:1434] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260320 02:59:01.935586 369 tablet_bootstrap.cc:492] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: No bootstrap required, opened a new log
I20260320 02:59:01.935722 369 ts_tablet_manager.cc:1403] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260320 02:59:01.936260 369 raft_consensus.cc:359] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:01.936368 369 raft_consensus.cc:385] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:01.936400 369 raft_consensus.cc:740] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32aeba715c645cb97a174f435a187e5, State: Initialized, Role: FOLLOWER
I20260320 02:59:01.936548 369 consensus_queue.cc:260] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:01.937124 369 ts_tablet_manager.cc:1434] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260320 02:59:02.018878 32524 tablet.cc:2379] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:59:02.055080 387 raft_consensus.cc:493] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:59:02.055261 387 raft_consensus.cc:515] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.055786 387 leader_election.cc:290] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0869b28899a543bf816956fd5667093d (127.23.132.65:41945), a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:02.056571 32476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3887a4fce9164d289e61d4243b6b87ac" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d" is_pre_election: true
I20260320 02:59:02.056565 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3887a4fce9164d289e61d4243b6b87ac" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5" is_pre_election: true
I20260320 02:59:02.056736 32476 raft_consensus.cc:2468] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:02.056785 32619 raft_consensus.cc:2468] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:02.057183 32695 leader_election.cc:304] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a0c8fe59e39f4fecb14c1a0fd79373b7; no voters:
I20260320 02:59:02.058365 387 raft_consensus.cc:2804] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:59:02.058681 387 raft_consensus.cc:493] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:59:02.059170 387 raft_consensus.cc:3060] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.060487 387 raft_consensus.cc:515] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.060989 387 leader_election.cc:290] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Requested vote from peers 0869b28899a543bf816956fd5667093d (127.23.132.65:41945), a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:02.061407 32476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3887a4fce9164d289e61d4243b6b87ac" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d"
I20260320 02:59:02.061406 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3887a4fce9164d289e61d4243b6b87ac" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5"
I20260320 02:59:02.061545 32619 raft_consensus.cc:3060] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.061545 32476 raft_consensus.cc:3060] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.062639 32619 raft_consensus.cc:2468] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:02.062639 32476 raft_consensus.cc:2468] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:02.063105 32693 leader_election.cc:304] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c8fe59e39f4fecb14c1a0fd79373b7, a32aeba715c645cb97a174f435a187e5; no voters:
I20260320 02:59:02.063472 387 raft_consensus.cc:2804] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:02.063580 387 raft_consensus.cc:697] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Becoming Leader. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Running, Role: LEADER
I20260320 02:59:02.063722 387 consensus_queue.cc:237] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.066754 32318 catalog_manager.cc:5671] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } health_report { overall_health: HEALTHY } } }
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975540, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.126@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:02.126838 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.116071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45767 (local address 127.0.0.1:56718)
0320 02:59:02.116366 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.116380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.116497 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.117231 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.117234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.117866 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:02.118112 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.118121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.119239 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.119242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.119616 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.119622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.119755 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.120342 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:02.120356 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:02.122149 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:02.124360 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:02.124370 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:02.124374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:02.124570 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:02.124777 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:02.124780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:02.124782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:02.124825 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:02.125772 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:02.125781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:02.125939 (+ 158us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.126060 (+ 121us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.126418 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20260320 02:59:02.126854 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.116561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.23.132.126:45767)
0320 02:59:02.117001 (+ 440us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.117005 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.117027 (+ 22us) server_negotiation.cc:408] Connection header received
0320 02:59:02.117071 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.117074 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.117119 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.117196 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:02.118233 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.118747 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.119741 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.119902 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.122274 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:02.122294 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:02.122296 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:02.122318 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:02.124247 (+ 1929us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:02.124659 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:02.124662 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:02.124664 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:02.124702 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:02.124899 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:02.124902 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:02.124904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:02.125276 (+ 372us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:02.125670 (+ 394us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.126139 (+ 469us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.126720 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":236,"server-negotiator.queue_time_us":344}
I20260320 02:59:02.132716 32318 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:56718:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260320 02:59:02.133695 32318 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:59:02.143011 32454 tablet_service.cc:1511] Processing CreateTablet for tablet 4718c6a3d6c94bb28d3891b99297898a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32afb4a20f7a4d80a6b15eacb71be05d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:59:02.143360 32741 tablet_service.cc:1511] Processing CreateTablet for tablet 4718c6a3d6c94bb28d3891b99297898a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32afb4a20f7a4d80a6b15eacb71be05d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:59:02.143874 32741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4718c6a3d6c94bb28d3891b99297898a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.144243 32454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4718c6a3d6c94bb28d3891b99297898a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.147570 370 tablet_bootstrap.cc:492] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: Bootstrap starting.
I20260320 02:59:02.147913 32599 tablet_service.cc:1511] Processing CreateTablet for tablet 4718c6a3d6c94bb28d3891b99297898a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=32afb4a20f7a4d80a6b15eacb71be05d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260320 02:59:02.148232 32599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4718c6a3d6c94bb28d3891b99297898a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.148373 368 tablet_bootstrap.cc:492] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: Bootstrap starting.
I20260320 02:59:02.148893 370 tablet_bootstrap.cc:654] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.150112 368 tablet_bootstrap.cc:654] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.152191 368 tablet_bootstrap.cc:492] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: No bootstrap required, opened a new log
I20260320 02:59:02.152290 368 ts_tablet_manager.cc:1403] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:02.152513 369 tablet_bootstrap.cc:492] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: Bootstrap starting.
I20260320 02:59:02.152791 368 raft_consensus.cc:359] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.152908 368 raft_consensus.cc:385] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.152940 368 raft_consensus.cc:740] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0869b28899a543bf816956fd5667093d, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.153083 368 consensus_queue.cc:260] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.153364 368 ts_tablet_manager.cc:1434] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260320 02:59:02.153462 370 tablet_bootstrap.cc:492] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: No bootstrap required, opened a new log
I20260320 02:59:02.153412 375 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.153539 375 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.153546 370 ts_tablet_manager.cc:1403] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260320 02:59:02.153999 369 tablet_bootstrap.cc:654] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.154744 370 raft_consensus.cc:359] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.155161 370 raft_consensus.cc:385] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.155231 370 raft_consensus.cc:740] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.155457 370 consensus_queue.cc:260] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.155963 370 ts_tablet_manager.cc:1434] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260320 02:59:02.156431 369 tablet_bootstrap.cc:492] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: No bootstrap required, opened a new log
I20260320 02:59:02.156435 387 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.156538 369 ts_tablet_manager.cc:1403] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260320 02:59:02.156551 387 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.157156 369 raft_consensus.cc:359] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.157312 369 raft_consensus.cc:385] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.157347 369 raft_consensus.cc:740] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32aeba715c645cb97a174f435a187e5, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.157513 369 consensus_queue.cc:260] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.158186 374 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.158332 374 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.158586 369 ts_tablet_manager.cc:1434] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260320 02:59:02.163681 340 tablet.cc:2379] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:59:02.199777 387 raft_consensus.cc:493] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:59:02.199941 387 raft_consensus.cc:515] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.200508 387 leader_election.cc:290] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0869b28899a543bf816956fd5667093d (127.23.132.65:41945), a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:02.201064 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4718c6a3d6c94bb28d3891b99297898a" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5" is_pre_election: true
I20260320 02:59:02.201117 32476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4718c6a3d6c94bb28d3891b99297898a" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d" is_pre_election: true
I20260320 02:59:02.201200 32619 raft_consensus.cc:2468] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:02.201264 32476 raft_consensus.cc:2468] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:02.201664 32693 leader_election.cc:304] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c8fe59e39f4fecb14c1a0fd79373b7, a32aeba715c645cb97a174f435a187e5; no voters:
I20260320 02:59:02.201882 387 raft_consensus.cc:2804] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:59:02.201936 387 raft_consensus.cc:493] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:59:02.201968 387 raft_consensus.cc:3060] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.203307 387 raft_consensus.cc:515] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.203675 387 leader_election.cc:290] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Requested vote from peers 0869b28899a543bf816956fd5667093d (127.23.132.65:41945), a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:02.204221 32476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4718c6a3d6c94bb28d3891b99297898a" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d"
I20260320 02:59:02.204377 32476 raft_consensus.cc:3060] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.204059 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4718c6a3d6c94bb28d3891b99297898a" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5"
I20260320 02:59:02.204527 32619 raft_consensus.cc:3060] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.205372 32476 raft_consensus.cc:2468] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:02.205508 32619 raft_consensus.cc:2468] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:02.205780 32695 leader_election.cc:304] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a0c8fe59e39f4fecb14c1a0fd79373b7; no voters:
I20260320 02:59:02.205986 387 raft_consensus.cc:2804] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:02.206070 387 raft_consensus.cc:697] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Becoming Leader. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Running, Role: LEADER
I20260320 02:59:02.206558 387 consensus_queue.cc:237] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.207876 378 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: TxnStatusTablet state changed. Reason: New leader a0c8fe59e39f4fecb14c1a0fd79373b7. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.208081 378 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:59:02.208410 408 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260320 02:59:02.208585 408 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260320 02:59:02.209383 32317 catalog_manager.cc:5671] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } health_report { overall_health: HEALTHY } } }
I20260320 02:59:02.239444 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.235906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:38205 (local address 127.0.0.1:49090)
0320 02:59:02.236087 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.236103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.236202 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.236582 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.236585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.236594 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:59:02.236797 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.236802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.237973 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.237977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.238404 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.238413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.238512 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.238558 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.239163 (+ 605us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.239179 (+ 16us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.239244 (+ 65us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.239318 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260320 02:59:02.239475 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.236034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.23.132.67:38205)
0320 02:59:02.236174 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.236179 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.236196 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:59:02.236310 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.236314 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.236372 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.236463 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:02.236906 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.237681 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.238518 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.238716 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.238763 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.238856 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.239165 (+ 309us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.239277 (+ 112us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.239334 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260320 02:59:02.242566 32476 raft_consensus.cc:1275] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.242568 32619 raft_consensus.cc:1275] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.243074 378 consensus_queue.cc:1048] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.243523 387 consensus_queue.cc:1048] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.245131 375 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: TxnStatusTablet state changed. Reason: New leader a0c8fe59e39f4fecb14c1a0fd79373b7. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.245244 375 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.245271 374 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: TxnStatusTablet state changed. Reason: New leader a0c8fe59e39f4fecb14c1a0fd79373b7. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.245393 374 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.247296 381 mvcc.cc:204] Tried to move back new op lower bound from 7266203821021204480 to 7266203820880080896. Current Snapshot: MvccSnapshot[applied={T|T < 7266203821021204480}]
I20260320 02:59:02.247421 387 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.247521 387 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:59:02.247586 376 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.247684 376 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a0c8fe59e39f4fecb14c1a0fd79373b7: This TxnStatusTablet replica's current role is: LEADER
I20260320 02:59:02.248746 375 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.248858 375 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P 0869b28899a543bf816956fd5667093d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.250881 374 tablet_replica.cc:442] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } } }
I20260320 02:59:02.251003 374 tablet_replica.cc:445] T 4718c6a3d6c94bb28d3891b99297898a P a32aeba715c645cb97a174f435a187e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260320 02:59:02.269243 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.264109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:38205 (local address 127.23.132.67:52139)
0320 02:59:02.264439 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.264476 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.264637 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.265014 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.265019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.265035 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:59:02.265557 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.265570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.267235 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.267241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.267965 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.267978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.268224 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.268300 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.268836 (+ 536us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.268842 (+ 6us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.268915 (+ 73us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.269026 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":142,"threads_started":1}
I20260320 02:59:02.269295 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.264167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:52139 (local address 127.23.132.67:38205)
0320 02:59:02.264301 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.264306 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.264687 (+ 381us) server_negotiation.cc:408] Connection header received
0320 02:59:02.264740 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.264743 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.264793 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.264860 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:02.265763 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.267032 (+ 1269us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.268309 (+ 1277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.268566 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.268628 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.268732 (+ 104us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.268813 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.268976 (+ 163us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.269129 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260320 02:59:02.279944 32476 raft_consensus.cc:1275] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.280018 32619 raft_consensus.cc:1275] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.280589 376 consensus_queue.cc:1048] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.280871 378 consensus_queue.cc:1048] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.340582 32317 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:02.340916 32317 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:02.344777 32317 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Sending DeleteTablet for 3 replicas of tablet 3887a4fce9164d289e61d4243b6b87ac
I20260320 02:59:02.346006 32454 tablet_service.cc:1558] Processing DeleteTablet for tablet 3887a4fce9164d289e61d4243b6b87ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:51976
I20260320 02:59:02.346421 393 tablet_replica.cc:333] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: stopping tablet replica
I20260320 02:59:02.346545 393 raft_consensus.cc:2243] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:02.346733 393 raft_consensus.cc:2272] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.346112 32741 tablet_service.cc:1558] Processing DeleteTablet for tablet 3887a4fce9164d289e61d4243b6b87ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:49078
I20260320 02:59:02.347949 393 ts_tablet_manager.cc:1916] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.349951 392 tablet_replica.cc:333] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: stopping tablet replica
I20260320 02:59:02.350090 392 raft_consensus.cc:2243] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:59:02.350490 392 raft_consensus.cc:2272] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.350555 393 ts_tablet_manager.cc:1929] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:59:02.350817 393 log.cc:1199] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/3887a4fce9164d289e61d4243b6b87ac
I20260320 02:59:02.351076 393 ts_tablet_manager.cc:1950] T 3887a4fce9164d289e61d4243b6b87ac P 0869b28899a543bf816956fd5667093d: Deleting consensus metadata
I20260320 02:59:02.351655 392 ts_tablet_manager.cc:1916] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.354169 392 ts_tablet_manager.cc:1929] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:59:02.354310 392 log.cc:1199] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/3887a4fce9164d289e61d4243b6b87ac
I20260320 02:59:02.354553 392 ts_tablet_manager.cc:1950] T 3887a4fce9164d289e61d4243b6b87ac P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting consensus metadata
I20260320 02:59:02.355127 32305 catalog_manager.cc:5002] TS 0869b28899a543bf816956fd5667093d (127.23.132.65:41945): tablet 3887a4fce9164d289e61d4243b6b87ac (table test-table [id=ce46c7a442d6437dbd84fa37f5e3c85f]) successfully deleted
I20260320 02:59:02.355755 32304 catalog_manager.cc:5002] TS a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205): tablet 3887a4fce9164d289e61d4243b6b87ac (table test-table [id=ce46c7a442d6437dbd84fa37f5e3c85f]) successfully deleted
I20260320 02:59:02.358386 32599 tablet_service.cc:1558] Processing DeleteTablet for tablet 3887a4fce9164d289e61d4243b6b87ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:40554
I20260320 02:59:02.358574 391 tablet_replica.cc:333] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: stopping tablet replica
I20260320 02:59:02.358683 391 raft_consensus.cc:2243] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:02.358798 391 raft_consensus.cc:2272] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.359683 391 ts_tablet_manager.cc:1916] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.362738 391 ts_tablet_manager.cc:1929] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:59:02.362819 391 log.cc:1199] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/3887a4fce9164d289e61d4243b6b87ac
I20260320 02:59:02.363035 391 ts_tablet_manager.cc:1950] T 3887a4fce9164d289e61d4243b6b87ac P a32aeba715c645cb97a174f435a187e5: Deleting consensus metadata
I20260320 02:59:02.367028 32303 catalog_manager.cc:5002] TS a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383): tablet 3887a4fce9164d289e61d4243b6b87ac (table test-table [id=ce46c7a442d6437dbd84fa37f5e3c85f]) successfully deleted
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[32270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:02.394796 32367 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:02.385437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.23.132.126:45767)
0320 02:59:02.385652 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.385658 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.386150 (+ 492us) server_negotiation.cc:408] Connection header received
0320 02:59:02.386411 (+ 261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.386415 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.386489 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.386571 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:02.387848 (+ 1277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.389196 (+ 1348us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.390299 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.390495 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.393984 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56720: BlockingRecv error: recv got EOF from 127.0.0.1:56720 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[32270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM
I20260320 02:59:02.407744 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.397312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.23.132.126:45767)
0320 02:59:02.397480 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.397483 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.397498 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:59:02.397538 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.397540 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.397585 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.397994 (+ 409us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:02.398764 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.399258 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.399852 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.400026 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.402733 (+ 2707us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:02.402774 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:02.402777 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:02.402828 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:02.405362 (+ 2534us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:02.405980 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:02.405984 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:02.405986 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:02.406095 (+ 109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:02.406490 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:02.406493 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:02.406495 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:02.406695 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:02.406828 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.407239 (+ 411us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.407425 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260320 02:59:02.411113 32317 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:59:02.411870 32317 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:59:02.418998 32454 tablet_service.cc:1511] Processing CreateTablet for tablet 28944e02fe224baa9dc34e2e581211fc (DEFAULT_TABLE table=test-table [id=cb15887678bf4a6f8f1735994d5e2739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:02.419195 32741 tablet_service.cc:1511] Processing CreateTablet for tablet 28944e02fe224baa9dc34e2e581211fc (DEFAULT_TABLE table=test-table [id=cb15887678bf4a6f8f1735994d5e2739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:02.419359 32454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28944e02fe224baa9dc34e2e581211fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.419476 32741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28944e02fe224baa9dc34e2e581211fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.421707 370 tablet_bootstrap.cc:492] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Bootstrap starting.
I20260320 02:59:02.422346 368 tablet_bootstrap.cc:492] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Bootstrap starting.
I20260320 02:59:02.422626 32599 tablet_service.cc:1511] Processing CreateTablet for tablet 28944e02fe224baa9dc34e2e581211fc (DEFAULT_TABLE table=test-table [id=cb15887678bf4a6f8f1735994d5e2739]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:02.422991 32599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28944e02fe224baa9dc34e2e581211fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.423863 368 tablet_bootstrap.cc:654] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.426175 368 tablet_bootstrap.cc:492] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: No bootstrap required, opened a new log
I20260320 02:59:02.426304 368 ts_tablet_manager.cc:1403] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:02.427047 370 tablet_bootstrap.cc:654] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.426996 368 raft_consensus.cc:359] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.427126 368 raft_consensus.cc:385] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.427165 368 raft_consensus.cc:740] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0869b28899a543bf816956fd5667093d, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.427464 368 consensus_queue.cc:260] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.427933 368 ts_tablet_manager.cc:1434] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260320 02:59:02.428179 369 tablet_bootstrap.cc:492] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Bootstrap starting.
I20260320 02:59:02.429322 369 tablet_bootstrap.cc:654] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.431277 370 tablet_bootstrap.cc:492] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: No bootstrap required, opened a new log
I20260320 02:59:02.431389 370 ts_tablet_manager.cc:1403] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260320 02:59:02.431881 370 raft_consensus.cc:359] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.432022 370 raft_consensus.cc:385] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.432061 370 raft_consensus.cc:740] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.432192 370 consensus_queue.cc:260] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.432482 370 ts_tablet_manager.cc:1434] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260320 02:59:02.433079 369 tablet_bootstrap.cc:492] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: No bootstrap required, opened a new log
I20260320 02:59:02.433177 369 ts_tablet_manager.cc:1403] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260320 02:59:02.433665 369 raft_consensus.cc:359] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.434624 369 raft_consensus.cc:385] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.434669 369 raft_consensus.cc:740] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32aeba715c645cb97a174f435a187e5, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.434809 369 consensus_queue.cc:260] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.435153 369 ts_tablet_manager.cc:1434] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260320 02:59:02.563151 376 raft_consensus.cc:493] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:59:02.563298 376 raft_consensus.cc:515] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.563827 376 leader_election.cc:290] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0869b28899a543bf816956fd5667093d (127.23.132.65:41945), a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:02.564405 32476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28944e02fe224baa9dc34e2e581211fc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d" is_pre_election: true
I20260320 02:59:02.564602 32476 raft_consensus.cc:2468] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:02.565074 32695 leader_election.cc:304] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a0c8fe59e39f4fecb14c1a0fd79373b7; no voters:
I20260320 02:59:02.565361 376 raft_consensus.cc:2804] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:59:02.565342 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28944e02fe224baa9dc34e2e581211fc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5" is_pre_election: true
I20260320 02:59:02.565418 376 raft_consensus.cc:493] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:59:02.565450 376 raft_consensus.cc:3060] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.565497 32619 raft_consensus.cc:2468] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 0.
I20260320 02:59:02.566915 376 raft_consensus.cc:515] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.567425 376 leader_election.cc:290] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Requested vote from peers 0869b28899a543bf816956fd5667093d (127.23.132.65:41945), a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383)
I20260320 02:59:02.567778 32476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28944e02fe224baa9dc34e2e581211fc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0869b28899a543bf816956fd5667093d"
I20260320 02:59:02.568135 32476 raft_consensus.cc:3060] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.568104 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28944e02fe224baa9dc34e2e581211fc" candidate_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5"
I20260320 02:59:02.568229 32619 raft_consensus.cc:3060] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.569320 32476 raft_consensus.cc:2468] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:02.569320 32619 raft_consensus.cc:2468] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c8fe59e39f4fecb14c1a0fd79373b7 in term 1.
I20260320 02:59:02.569727 32695 leader_election.cc:304] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a0c8fe59e39f4fecb14c1a0fd79373b7; no voters:
I20260320 02:59:02.569996 376 raft_consensus.cc:2804] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:02.570091 376 raft_consensus.cc:697] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Becoming Leader. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Running, Role: LEADER
I20260320 02:59:02.570230 376 consensus_queue.cc:237] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.572365 32317 catalog_manager.cc:5671] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } health_report { overall_health: HEALTHY } } }
I20260320 02:59:02.598119 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.594185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.23.132.67:38205)
0320 02:59:02.594315 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.594318 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.594621 (+ 303us) server_negotiation.cc:408] Connection header received
0320 02:59:02.595031 (+ 410us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.595035 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.595264 (+ 229us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.595344 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:02.595903 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.596452 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.597271 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.597458 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.597510 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.597611 (+ 101us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.597681 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.597940 (+ 259us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.597997 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260320 02:59:02.600993 32476 raft_consensus.cc:1275] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.600984 32619 raft_consensus.cc:1275] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Refusing update from remote peer a0c8fe59e39f4fecb14c1a0fd79373b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.601450 376 consensus_queue.cc:1048] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.601728 376 consensus_queue.cc:1048] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.610569 32318 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:02.610882 32318 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:02.613420 32318 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Sending DeleteTablet for 3 replicas of tablet 28944e02fe224baa9dc34e2e581211fc
I20260320 02:59:02.614382 32454 tablet_service.cc:1558] Processing DeleteTablet for tablet 28944e02fe224baa9dc34e2e581211fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:51976
I20260320 02:59:02.614843 393 tablet_replica.cc:333] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: stopping tablet replica
I20260320 02:59:02.615002 393 raft_consensus.cc:2243] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:02.614900 32599 tablet_service.cc:1558] Processing DeleteTablet for tablet 28944e02fe224baa9dc34e2e581211fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:40554
I20260320 02:59:02.615160 393 raft_consensus.cc:2272] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.615154 32741 tablet_service.cc:1558] Processing DeleteTablet for tablet 28944e02fe224baa9dc34e2e581211fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:49078
I20260320 02:59:02.615193 391 tablet_replica.cc:333] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: stopping tablet replica
I20260320 02:59:02.615302 391 raft_consensus.cc:2243] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:02.615334 392 tablet_replica.cc:333] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: stopping tablet replica
I20260320 02:59:02.615434 391 raft_consensus.cc:2272] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.615438 392 raft_consensus.cc:2243] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:59:02.615633 392 raft_consensus.cc:2272] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.616362 391 ts_tablet_manager.cc:1916] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.616364 393 ts_tablet_manager.cc:1916] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.616300 32318 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260320 02:59:02.616461 392 ts_tablet_manager.cc:1916] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260320 02:59:02.617558 32318 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:59:02.619081 393 ts_tablet_manager.cc:1929] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:02.619153 393 log.cc:1199] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-0/wal/wals/28944e02fe224baa9dc34e2e581211fc
I20260320 02:59:02.619146 392 ts_tablet_manager.cc:1929] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:02.619201 392 log.cc:1199] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/28944e02fe224baa9dc34e2e581211fc
I20260320 02:59:02.619393 393 ts_tablet_manager.cc:1950] T 28944e02fe224baa9dc34e2e581211fc P 0869b28899a543bf816956fd5667093d: Deleting consensus metadata
I20260320 02:59:02.619413 392 ts_tablet_manager.cc:1950] T 28944e02fe224baa9dc34e2e581211fc P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting consensus metadata
I20260320 02:59:02.621001 32304 catalog_manager.cc:5002] TS a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205): tablet 28944e02fe224baa9dc34e2e581211fc (table test-table [id=cb15887678bf4a6f8f1735994d5e2739]) successfully deleted
I20260320 02:59:02.621206 32305 catalog_manager.cc:5002] TS 0869b28899a543bf816956fd5667093d (127.23.132.65:41945): tablet 28944e02fe224baa9dc34e2e581211fc (table test-table [id=cb15887678bf4a6f8f1735994d5e2739]) successfully deleted
I20260320 02:59:02.621551 391 ts_tablet_manager.cc:1929] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:02.621665 391 log.cc:1199] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/28944e02fe224baa9dc34e2e581211fc
I20260320 02:59:02.622400 391 ts_tablet_manager.cc:1950] T 28944e02fe224baa9dc34e2e581211fc P a32aeba715c645cb97a174f435a187e5: Deleting consensus metadata
I20260320 02:59:02.623219 32303 catalog_manager.cc:5002] TS a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383): tablet 28944e02fe224baa9dc34e2e581211fc (table test-table [id=cb15887678bf4a6f8f1735994d5e2739]) successfully deleted
I20260320 02:59:02.627156 32599 tablet_service.cc:1511] Processing CreateTablet for tablet 958ddb4821cb43b0854666e1b879b91e (DEFAULT_TABLE table=test-table [id=b94729c302f8454e941dbaccec3f464b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:02.627393 32741 tablet_service.cc:1511] Processing CreateTablet for tablet 958ddb4821cb43b0854666e1b879b91e (DEFAULT_TABLE table=test-table [id=b94729c302f8454e941dbaccec3f464b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:02.627547 32454 tablet_service.cc:1511] Processing CreateTablet for tablet 958ddb4821cb43b0854666e1b879b91e (DEFAULT_TABLE table=test-table [id=b94729c302f8454e941dbaccec3f464b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:02.627714 32599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 958ddb4821cb43b0854666e1b879b91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.627908 32741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 958ddb4821cb43b0854666e1b879b91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.628232 32454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 958ddb4821cb43b0854666e1b879b91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:02.630600 369 tablet_bootstrap.cc:492] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Bootstrap starting.
I20260320 02:59:02.630640 370 tablet_bootstrap.cc:492] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Bootstrap starting.
I20260320 02:59:02.630952 368 tablet_bootstrap.cc:492] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: Bootstrap starting.
I20260320 02:59:02.632062 369 tablet_bootstrap.cc:654] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.632951 370 tablet_bootstrap.cc:654] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.633256 369 tablet_bootstrap.cc:492] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: No bootstrap required, opened a new log
I20260320 02:59:02.633280 368 tablet_bootstrap.cc:654] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:02.633322 369 ts_tablet_manager.cc:1403] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260320 02:59:02.633818 369 raft_consensus.cc:359] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.634442 369 raft_consensus.cc:385] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.634521 369 raft_consensus.cc:740] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32aeba715c645cb97a174f435a187e5, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.634835 368 tablet_bootstrap.cc:492] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: No bootstrap required, opened a new log
I20260320 02:59:02.634948 368 ts_tablet_manager.cc:1403] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:02.635373 368 raft_consensus.cc:359] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.635473 368 raft_consensus.cc:385] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.635502 368 raft_consensus.cc:740] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0869b28899a543bf816956fd5667093d, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.635555 370 tablet_bootstrap.cc:492] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: No bootstrap required, opened a new log
I20260320 02:59:02.635633 368 consensus_queue.cc:260] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.635718 370 ts_tablet_manager.cc:1403] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260320 02:59:02.636324 368 ts_tablet_manager.cc:1434] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260320 02:59:02.637326 369 consensus_queue.cc:260] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.637491 370 raft_consensus.cc:359] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.637701 370 raft_consensus.cc:385] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:02.637925 369 ts_tablet_manager.cc:1434] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260320 02:59:02.637925 370 raft_consensus.cc:740] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c8fe59e39f4fecb14c1a0fd79373b7, State: Initialized, Role: FOLLOWER
I20260320 02:59:02.638168 370 consensus_queue.cc:260] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.638522 370 ts_tablet_manager.cc:1434] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260320 02:59:02.661147 375 raft_consensus.cc:493] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:59:02.661377 375 raft_consensus.cc:515] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.662796 375 leader_election.cc:290] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383), a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205)
I20260320 02:59:02.667099 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.663003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:36383 (local address 127.23.132.65:40429)
0320 02:59:02.663316 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.663332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.663443 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.663819 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.663822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.663841 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:02.664115 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.664123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.665430 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.665435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.666426 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.666435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.666555 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.666580 (+ 25us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.666658 (+ 78us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.666720 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":73,"threads_started":1}
I20260320 02:59:02.668087 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.663138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:40429 (local address 127.23.132.66:36383)
0320 02:59:02.663378 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.663382 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.663398 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:02.663568 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.663572 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.663625 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.663717 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:02.664236 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.665314 (+ 1078us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.666567 (+ 1253us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.667260 (+ 693us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.667304 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.667359 (+ 55us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.667809 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260320 02:59:02.668457 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.663512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:37181 (local address 127.23.132.67:38205)
0320 02:59:02.663654 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.663659 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.663850 (+ 191us) server_negotiation.cc:408] Connection header received
0320 02:59:02.664053 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.664057 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.664103 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.664181 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:02.665145 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.665854 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.667036 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.667527 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.667562 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.668149 (+ 587us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.668336 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260320 02:59:02.669203 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.663161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:38205 (local address 127.23.132.65:37181)
0320 02:59:02.663748 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.663763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.663952 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.664571 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.664575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.664600 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:02.664872 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.664880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.665961 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.665964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.666896 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.666907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.667757 (+ 850us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.667820 (+ 63us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.668977 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.669058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"spinlock_wait_cycles":1408,"thread_start_us":74,"threads_started":1}
I20260320 02:59:02.669260 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "958ddb4821cb43b0854666e1b879b91e" candidate_uuid: "0869b28899a543bf816956fd5667093d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5" is_pre_election: true
I20260320 02:59:02.669425 32619 raft_consensus.cc:2468] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0869b28899a543bf816956fd5667093d in term 0.
I20260320 02:59:02.670032 32409 leader_election.cc:304] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a32aeba715c645cb97a174f435a187e5; no voters:
I20260320 02:59:02.670096 32758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "958ddb4821cb43b0854666e1b879b91e" candidate_uuid: "0869b28899a543bf816956fd5667093d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" is_pre_election: true
I20260320 02:59:02.670281 32758 raft_consensus.cc:2468] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0869b28899a543bf816956fd5667093d in term 0.
I20260320 02:59:02.670769 375 raft_consensus.cc:2804] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:59:02.670830 375 raft_consensus.cc:493] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:59:02.670962 375 raft_consensus.cc:3060] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.672037 375 raft_consensus.cc:515] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.672561 375 leader_election.cc:290] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [CANDIDATE]: Term 1 election: Requested vote from peers a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383), a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205)
I20260320 02:59:02.673586 32619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "958ddb4821cb43b0854666e1b879b91e" candidate_uuid: "0869b28899a543bf816956fd5667093d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a32aeba715c645cb97a174f435a187e5"
I20260320 02:59:02.673780 32619 raft_consensus.cc:3060] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.673744 32758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "958ddb4821cb43b0854666e1b879b91e" candidate_uuid: "0869b28899a543bf816956fd5667093d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7"
I20260320 02:59:02.673910 32758 raft_consensus.cc:3060] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:02.674804 32619 raft_consensus.cc:2468] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0869b28899a543bf816956fd5667093d in term 1.
I20260320 02:59:02.675014 32758 raft_consensus.cc:2468] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0869b28899a543bf816956fd5667093d in term 1.
I20260320 02:59:02.675522 32410 leader_election.cc:304] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869b28899a543bf816956fd5667093d, a0c8fe59e39f4fecb14c1a0fd79373b7; no voters:
I20260320 02:59:02.675951 375 raft_consensus.cc:2804] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:02.676142 375 raft_consensus.cc:697] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 1 LEADER]: Becoming Leader. State: Replica: 0869b28899a543bf816956fd5667093d, State: Running, Role: LEADER
I20260320 02:59:02.677445 375 consensus_queue.cc:237] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } }
I20260320 02:59:02.683128 32317 catalog_manager.cc:5671] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0869b28899a543bf816956fd5667093d (127.23.132.65). New cstate: current_term: 1 leader_uuid: "0869b28899a543bf816956fd5667093d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869b28899a543bf816956fd5667093d" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 41945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 } health_report { overall_health: UNKNOWN } } }
I20260320 02:59:02.716027 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.711107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.23.132.65:41945)
0320 02:59:02.711395 (+ 288us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.711398 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.711785 (+ 387us) server_negotiation.cc:408] Connection header received
0320 02:59:02.712159 (+ 374us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.712163 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.712246 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.712359 (+ 113us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:02.713074 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.713734 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.714793 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.715017 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.715080 (+ 63us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.715562 (+ 482us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.715669 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.715765 (+ 96us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.715826 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
I20260320 02:59:02.730510 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.727334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:56565 (local address 127.23.132.67:38205)
0320 02:59:02.727585 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.727590 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.727796 (+ 206us) server_negotiation.cc:408] Connection header received
0320 02:59:02.727944 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.727948 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.727994 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.728060 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:02.728464 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.729140 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.729844 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.730027 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.730069 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.730143 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.730202 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.730312 (+ 110us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.730364 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260320 02:59:02.730522 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.727166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:38205 (local address 127.23.132.65:56565)
0320 02:59:02.727702 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.727716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.727837 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.728085 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.728088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.728096 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:59:02.728356 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.728363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.729264 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.729268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.729718 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.729725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.729854 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.729905 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.730231 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.730236 (+ 5us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.730291 (+ 55us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.730369 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":139,"threads_started":1}
I20260320 02:59:02.740430 32619 raft_consensus.cc:1275] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Refusing update from remote peer 0869b28899a543bf816956fd5667093d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.740517 32758 raft_consensus.cc:1275] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Refusing update from remote peer 0869b28899a543bf816956fd5667093d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:02.741372 446 consensus_queue.cc:1048] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0c8fe59e39f4fecb14c1a0fd79373b7" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 38205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.741587 375 consensus_queue.cc:1048] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a32aeba715c645cb97a174f435a187e5" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:02.765947 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.760767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:41945 (local address 127.23.132.67:40275)
0320 02:59:02.761062 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.761087 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:02.761171 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:02.762269 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:02.762273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:02.762282 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0320 02:59:02.762688 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.762756 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.763770 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.763774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:02.764361 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:02.764371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.764483 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.764511 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.765563 (+ 1052us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.765571 (+ 8us) client_negotiation.cc:769] Sending connection context
0320 02:59:02.765658 (+ 87us) client_negotiation.cc:240] Negotiation successful
0320 02:59:02.765749 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260320 02:59:02.766120 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.761655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:40275 (local address 127.23.132.65:41945)
0320 02:59:02.761845 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:02.761848 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:02.761859 (+ 11us) server_negotiation.cc:408] Connection header received
0320 02:59:02.761893 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:02.761896 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:02.761948 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:02.762020 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:02.762884 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.763581 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:02.764842 (+ 1261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:02.765079 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:02.765135 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:02.765303 (+ 168us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:02.765403 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:02.765928 (+ 525us) server_negotiation.cc:300] Negotiation successful
0320 02:59:02.765993 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260320 02:59:02.799356 32316 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:02.799551 32316 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:02.803268 32316 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 54fa41b6692a406c8e42b873db738d1e: Sending DeleteTablet for 3 replicas of tablet 958ddb4821cb43b0854666e1b879b91e
I20260320 02:59:02.804741 32741 tablet_service.cc:1558] Processing DeleteTablet for tablet 958ddb4821cb43b0854666e1b879b91e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:49078
I20260320 02:59:02.804745 32599 tablet_service.cc:1558] Processing DeleteTablet for tablet 958ddb4821cb43b0854666e1b879b91e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:40554
I20260320 02:59:02.804741 32454 tablet_service.cc:1558] Processing DeleteTablet for tablet 958ddb4821cb43b0854666e1b879b91e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:02 UTC) from {username='oryx', principal='oryx/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:51976
I20260320 02:59:02.804983 392 tablet_replica.cc:333] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: stopping tablet replica
I20260320 02:59:02.805112 392 raft_consensus.cc:2243] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:02.805336 392 raft_consensus.cc:2272] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.805553 393 tablet_replica.cc:333] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: stopping tablet replica
I20260320 02:59:02.805660 393 raft_consensus.cc:2243] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:59:02.806231 393 raft_consensus.cc:2272] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.806421 391 tablet_replica.cc:333] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: stopping tablet replica
I20260320 02:59:02.806555 391 raft_consensus.cc:2243] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:02.806739 391 raft_consensus.cc:2272] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:02.807647 393 ts_tablet_manager.cc:1916] T 958ddb4821cb43b0854666e1b879b91e P 0869b28899a543bf816956fd5667093d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.807647 392 ts_tablet_manager.cc:1916] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.809000 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 32392
I20260320 02:59:02.811249 391 ts_tablet_manager.cc:1916] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:02.811731 392 ts_tablet_manager.cc:1929] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:59:02.811985 392 log.cc:1199] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/958ddb4821cb43b0854666e1b879b91e
I20260320 02:59:02.812917 392 ts_tablet_manager.cc:1950] T 958ddb4821cb43b0854666e1b879b91e P a0c8fe59e39f4fecb14c1a0fd79373b7: Deleting consensus metadata
I20260320 02:59:02.814432 32304 catalog_manager.cc:5002] TS a0c8fe59e39f4fecb14c1a0fd79373b7 (127.23.132.67:38205): tablet 958ddb4821cb43b0854666e1b879b91e (table test-table [id=b94729c302f8454e941dbaccec3f464b]) successfully deleted
I20260320 02:59:02.816367 391 ts_tablet_manager.cc:1929] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260320 02:59:02.816486 391 log.cc:1199] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773975462196075-24081-0/minicluster-data/ts-1/wal/wals/958ddb4821cb43b0854666e1b879b91e
I20260320 02:59:02.817147 391 ts_tablet_manager.cc:1950] T 958ddb4821cb43b0854666e1b879b91e P a32aeba715c645cb97a174f435a187e5: Deleting consensus metadata
I20260320 02:59:02.819346 32303 catalog_manager.cc:5002] TS a32aeba715c645cb97a174f435a187e5 (127.23.132.66:36383): tablet 958ddb4821cb43b0854666e1b879b91e (table test-table [id=b94729c302f8454e941dbaccec3f464b]) successfully deleted
W20260320 02:59:02.823154 32305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.132.65:41945 (error 108)
I20260320 02:59:02.823335 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 32534
I20260320 02:59:02.826561 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:02.826170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:41945 (local address 127.0.0.1:51988)
0320 02:59:02.826395 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:02.826460 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.65:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
W20260320 02:59:02.826802 32305 catalog_manager.cc:4729] TS 0869b28899a543bf816956fd5667093d (127.23.132.65:41945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 958ddb4821cb43b0854666e1b879b91e: Network error: Client connection negotiation failed: client connection to 127.23.132.65:41945: connect: Connection refused (error 111)
I20260320 02:59:02.834486 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 32676
I20260320 02:59:02.843748 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 32286
2026-03-20T02:59:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4925 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[462](info): set up 2 sockets
Mar 20 02:59:02 dist-test-slave-v3hj krb5kdc[462](info): commencing operation
krb5kdc: starting...
W20260320 02:59:05.064817 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.193s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:59:05 dist-test-slave-v3hj krb5kdc[462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:59:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:05.266547 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:37117
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:37117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:05.406124 478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:05.406369 478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:05.406462 478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:05.409965 478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:05.410050 478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:05.410082 478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:05.410105 478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:05.410338 478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:05.419072 478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:37117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:37117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:05.421625 478 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:05.422593 478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:05.429608 483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:05.429607 486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:05.431017 484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:05.431073 478 server_base.cc:1061] running on GCE node
I20260320 02:59:05.433097 478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:05.435125 478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:05.436614 478 hybrid_clock.cc:648] HybridClock initialized: now 1773975545436551 us; error 85 us; skew 500 ppm
W20260320 02:59:05.442855 478 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-03-20T02:59:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2597 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:59:05 dist-test-slave-v3hj krb5kdc[491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:59:05 dist-test-slave-v3hj krb5kdc[491](info): set up 2 sockets
Mar 20 02:59:05 dist-test-slave-v3hj krb5kdc[491](info): commencing operation
krb5kdc: starting...
W20260320 02:59:07.519754 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.048s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:59:07 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:59:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:07.686344 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39437
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:07.795843 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:07.796097 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:07.796190 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:07.800073 507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:07.800163 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:07.800195 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:07.800222 507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:07.800251 507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:07.805428 507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:07.807456 507 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:07.808763 507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:07.817858 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:07.818172 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:07.818434 507 server_base.cc:1061] running on GCE node
W20260320 02:59:07.818575 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:07.819356 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:07.820493 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:07.821707 507 hybrid_clock.cc:648] HybridClock initialized: now 1773975547821745 us; error 99 us; skew 500 ppm
Mar 20 02:59:07 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975547, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:07.824648 507 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:59:07.825858 507 webserver.cc:492] Webserver started at http://127.23.132.126:37517/ using document root <none> and password file <none>
I20260320 02:59:07.827046 507 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:07.827121 507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:07.827441 507 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:07.829272 507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "59abe9003d844388a7d13a33bbce392d"
format_stamp: "Formatted at 2026-03-20 02:59:07 on dist-test-slave-v3hj"
server_key: "60b38d84bedb82807ea84a95fdbff78a"
server_key_iv: "a233c21e269b62561c797e97799dacbc"
server_key_version: "encryptionkey@0"
I20260320 02:59:07.829771 507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "59abe9003d844388a7d13a33bbce392d"
format_stamp: "Formatted at 2026-03-20 02:59:07 on dist-test-slave-v3hj"
server_key: "60b38d84bedb82807ea84a95fdbff78a"
server_key_iv: "a233c21e269b62561c797e97799dacbc"
server_key_version: "encryptionkey@0"
I20260320 02:59:07.833546 507 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:07.836241 522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:07.837323 507 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260320 02:59:07.837496 507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "59abe9003d844388a7d13a33bbce392d"
format_stamp: "Formatted at 2026-03-20 02:59:07 on dist-test-slave-v3hj"
server_key: "60b38d84bedb82807ea84a95fdbff78a"
server_key_iv: "a233c21e269b62561c797e97799dacbc"
server_key_version: "encryptionkey@0"
I20260320 02:59:07.837682 507 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:07.852552 507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:07.856184 507 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:07.856374 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:07.864253 507 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:39437
I20260320 02:59:07.864261 575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:39437 every 8 connection(s)
I20260320 02:59:07.865276 507 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:59:07.868029 576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:07.874040 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 507
I20260320 02:59:07.874325 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:59:07.874631 24081 external_mini_cluster.cc:1468] Setting key 4a99a7ae94f1a8aa548260bfd795dda0
I20260320 02:59:07.876539 576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Bootstrap starting.
I20260320 02:59:07.880319 576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:07.881413 576 log.cc:826] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Log is configured to *not* fsync() on all Append() calls
Mar 20 02:59:07 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:59:07.885166 576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: No bootstrap required, opened a new log
I20260320 02:59:07.888509 576 raft_consensus.cc:359] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:07.888737 576 raft_consensus.cc:385] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:07.888773 576 raft_consensus.cc:740] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59abe9003d844388a7d13a33bbce392d, State: Initialized, Role: FOLLOWER
I20260320 02:59:07.889298 576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:07.889452 576 raft_consensus.cc:399] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:59:07.889498 576 raft_consensus.cc:493] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:59:07.889597 576 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:07.891891 576 raft_consensus.cc:515] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:07.892180 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:07.876778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.23.132.126:39437)
0320 02:59:07.877197 (+ 419us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:07.877207 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:07.877239 (+ 32us) server_negotiation.cc:408] Connection header received
0320 02:59:07.877867 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:07.877911 (+ 44us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:07.878207 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:07.878484 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:07.880067 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:07.880976 (+ 909us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:07.882124 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:07.882464 (+ 340us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:07.885132 (+ 2668us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:07.885163 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:07.885177 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:07.885227 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:07.888081 (+ 2854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:07.888740 (+ 659us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:07.888750 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:07.888756 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:07.888975 (+ 219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:07.889397 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:07.889401 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:07.889402 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:07.890023 (+ 621us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:07.890208 (+ 185us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:07.891164 (+ 956us) server_negotiation.cc:300] Negotiation successful
0320 02:59:07.891423 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260320 02:59:07.892906 576 leader_election.cc:304] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59abe9003d844388a7d13a33bbce392d; no voters:
I20260320 02:59:07.893241 576 leader_election.cc:290] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:59:07.893363 581 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:07.893586 581 raft_consensus.cc:697] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 LEADER]: Becoming Leader. State: Replica: 59abe9003d844388a7d13a33bbce392d, State: Running, Role: LEADER
I20260320 02:59:07.894172 581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:07.894992 576 sys_catalog.cc:565] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:59:07.896080 582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59abe9003d844388a7d13a33bbce392d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } } }
I20260320 02:59:07.896284 582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:07.896756 590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:59:07.899014 583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59abe9003d844388a7d13a33bbce392d. Latest consensus state: current_term: 1 leader_uuid: "59abe9003d844388a7d13a33bbce392d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } } }
I20260320 02:59:07.899333 583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:07.900578 590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:59:07.907480 590 catalog_manager.cc:1357] Generated new cluster ID: bea302741a434b998baf9bbeabbc14a0
I20260320 02:59:07.907568 590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:59:07.919186 590 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:59:07.920449 590 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:59:07.937486 590 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Generated new TSK 0
I20260320 02:59:07.938453 590 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:08.006534 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39437
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:08.125463 604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:08.125696 604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:08.125775 604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:08.129372 604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:08.129462 604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:08.129542 604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:59:08.135051 604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:08.136212 604 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:08.137022 604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:08.144006 612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:08.144110 610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:08.144073 609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:08.145191 604 server_base.cc:1061] running on GCE node
I20260320 02:59:08.145560 604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:08.146178 604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:08.147351 604 hybrid_clock.cc:648] HybridClock initialized: now 1773975548147332 us; error 33 us; skew 500 ppm
Mar 20 02:59:08 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:08.150122 604 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:59:08.151273 604 webserver.cc:492] Webserver started at http://127.23.132.65:35901/ using document root <none> and password file <none>
I20260320 02:59:08.151965 604 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:08.152034 604 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:08.152314 604 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:08.154415 604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "71d7dba81b3f476cab08c05878da4918"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "6ea0b9fddbd22db50abb4307f39ff615"
server_key_iv: "865a9304962a38141d50f00e66d13022"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.154898 604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "71d7dba81b3f476cab08c05878da4918"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "6ea0b9fddbd22db50abb4307f39ff615"
server_key_iv: "865a9304962a38141d50f00e66d13022"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.159225 604 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:08.162045 619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.163146 604 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260320 02:59:08.163260 604 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "71d7dba81b3f476cab08c05878da4918"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "6ea0b9fddbd22db50abb4307f39ff615"
server_key_iv: "865a9304962a38141d50f00e66d13022"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.163368 604 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:08.175458 604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:08.179740 604 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:08.180104 604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:08.180802 604 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:08.181754 604 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:08.181847 604 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.181905 604 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:08.181932 604 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.196272 604 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:38887
I20260320 02:59:08.196375 733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:38887 every 8 connection(s)
I20260320 02:59:08.197594 604 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:59:08.205224 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 604
I20260320 02:59:08.205348 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:59:08.205595 24081 external_mini_cluster.cc:1468] Setting key 448a93d7f1f8079f2091692dd9b5dc3f
Mar 20 02:59:08 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:59:08.219561 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:08.200247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:39173)
0320 02:59:08.200709 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:08.200801 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:08.201595 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:08.202512 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:08.202521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:08.203596 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:08.204918 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:08.204936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.207623 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:08.207628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:08.208395 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:08.208408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.208663 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:08.209463 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:08.209503 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:08.212033 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:08.215310 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:08.215317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:08.215329 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:08.215586 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:08.216441 (+ 855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:08.216445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:08.216448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:08.216699 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:08.217597 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:08.217607 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:08.218436 (+ 829us) client_negotiation.cc:769] Sending connection context
0320 02:59:08.218653 (+ 217us) client_negotiation.cc:240] Negotiation successful
0320 02:59:08.218909 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260320 02:59:08.219805 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:08.200069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:39173 (local address 127.23.132.126:39437)
0320 02:59:08.200249 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:08.200253 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:08.200910 (+ 657us) server_negotiation.cc:408] Connection header received
0320 02:59:08.201996 (+ 1086us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:08.202011 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:08.202170 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:08.202253 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:08.205109 (+ 2856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.207228 (+ 2119us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:08.208554 (+ 1326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.208761 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:08.212230 (+ 3469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:08.212257 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:08.212261 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:08.212294 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:08.215108 (+ 2814us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:08.215801 (+ 693us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:08.215808 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:08.215810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:08.215927 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:08.217091 (+ 1164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:08.217096 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:08.217098 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:08.217340 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:08.217465 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:08.219164 (+ 1699us) server_negotiation.cc:300] Negotiation successful
0320 02:59:08.219301 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260320 02:59:08.221503 734 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39437
I20260320 02:59:08.222553 734 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:59:08.223264 734 heartbeater.cc:507] Master 127.23.132.126:39437 requested a full tablet report, sending...
I20260320 02:59:08.225018 539 ts_manager.cc:194] Registered new tserver with Master: 71d7dba81b3f476cab08c05878da4918 (127.23.132.65:38887)
I20260320 02:59:08.226521 539 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:39173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:08.279783 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39437
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:08.402654 741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:08.402904 741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:08.402964 741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:08.406813 741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:08.407035 741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:08.407158 741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:59:08.412631 741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:08.414535 741 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:08.415928 741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:08.424497 746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:08.424695 749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:08.424988 747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:08.425215 741 server_base.cc:1061] running on GCE node
I20260320 02:59:08.425740 741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:08.426354 741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:08.427525 741 hybrid_clock.cc:648] HybridClock initialized: now 1773975548427514 us; error 27 us; skew 500 ppm
Mar 20 02:59:08 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:08.430846 741 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:59:08.432193 741 webserver.cc:492] Webserver started at http://127.23.132.66:36643/ using document root <none> and password file <none>
I20260320 02:59:08.433176 741 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:08.433545 741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:08.433943 741 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:08.435951 741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "67e74fe9385e4978b3a23955a907d6e5"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "10b469e15262d1e0ab73eb133fcace67"
server_key_iv: "41614e773f23c3b857600efa54208382"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.436466 741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "67e74fe9385e4978b3a23955a907d6e5"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "10b469e15262d1e0ab73eb133fcace67"
server_key_iv: "41614e773f23c3b857600efa54208382"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.441221 741 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260320 02:59:08.444772 756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.446404 741 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260320 02:59:08.446527 741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "67e74fe9385e4978b3a23955a907d6e5"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "10b469e15262d1e0ab73eb133fcace67"
server_key_iv: "41614e773f23c3b857600efa54208382"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.446646 741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:08.471189 741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:08.475775 741 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:08.475991 741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:08.477108 741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:08.478549 741 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:08.478758 741 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.478858 741 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:08.478892 741 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.499718 741 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:33041
I20260320 02:59:08.499764 871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:33041 every 8 connection(s)
I20260320 02:59:08.501567 741 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:59:08.507740 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 741
I20260320 02:59:08.507856 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:59:08.508085 24081 external_mini_cluster.cc:1468] Setting key 3a9e43cb7848fbca8159c13915e0e44d
Mar 20 02:59:08 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:59:08.519109 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:08.503866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:44179 (local address 127.23.132.126:39437)
0320 02:59:08.504006 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:08.504009 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:08.504944 (+ 935us) server_negotiation.cc:408] Connection header received
0320 02:59:08.505827 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:08.505831 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:08.505892 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:08.505970 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:08.507797 (+ 1827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.508376 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:08.509335 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.509474 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:08.512904 (+ 3430us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:08.512925 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:08.512927 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:08.512954 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:08.515051 (+ 2097us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:08.516358 (+ 1307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:08.516363 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:08.516365 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:08.516446 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:08.517148 (+ 702us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:08.517152 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:08.517154 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:08.517352 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:08.517652 (+ 300us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:08.518698 (+ 1046us) server_negotiation.cc:300] Negotiation successful
0320 02:59:08.518859 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260320 02:59:08.519987 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:08.504310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:44179)
0320 02:59:08.504792 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:08.504827 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:08.505599 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:08.506130 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:08.506138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:08.506541 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:08.507542 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:08.507558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.508535 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:08.508539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:08.509054 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:08.509064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.509270 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:08.510231 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:08.510266 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:08.512689 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:08.515351 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:08.515360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:08.515372 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:08.516146 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:08.516718 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:08.516723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:08.516725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:08.516910 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:08.517652 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:08.517662 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:08.518397 (+ 735us) client_negotiation.cc:769] Sending connection context
0320 02:59:08.518683 (+ 286us) client_negotiation.cc:240] Negotiation successful
0320 02:59:08.518989 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":186,"threads_started":1}
I20260320 02:59:08.521831 872 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39437
I20260320 02:59:08.522066 872 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:08.523105 872 heartbeater.cc:507] Master 127.23.132.126:39437 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
I20260320 02:59:08.524778 539 ts_manager.cc:194] Registered new tserver with Master: 67e74fe9385e4978b3a23955a907d6e5 (127.23.132.66:33041)
I20260320 02:59:08.525426 539 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:44179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:08.592747 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39437
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:08.723414 879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:08.724359 879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:08.724447 879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:08.728603 879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:08.728696 879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:08.728782 879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:59:08.735797 879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:08.737591 879 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:08.739081 879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:08.746166 887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:08.746168 885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:08.747550 884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:08.748131 879 server_base.cc:1061] running on GCE node
I20260320 02:59:08.748973 879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:08.750249 879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:08.751458 879 hybrid_clock.cc:648] HybridClock initialized: now 1773975548751471 us; error 102 us; skew 500 ppm
Mar 20 02:59:08 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:08.755376 879 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:59:08.756528 879 webserver.cc:492] Webserver started at http://127.23.132.67:41135/ using document root <none> and password file <none>
I20260320 02:59:08.757068 879 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:08.757121 879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:08.757282 879 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:08.759100 879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "0637773bb2c3492e984e4fe8681330ff"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "050a74a2b0712593f47976fff7b96f2a"
server_key_iv: "9719aca15b9611b9cf6c0bcbfc04cafb"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.760188 879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "0637773bb2c3492e984e4fe8681330ff"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "050a74a2b0712593f47976fff7b96f2a"
server_key_iv: "9719aca15b9611b9cf6c0bcbfc04cafb"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.764253 879 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:59:08.767014 894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.768802 879 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260320 02:59:08.768924 879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "0637773bb2c3492e984e4fe8681330ff"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "050a74a2b0712593f47976fff7b96f2a"
server_key_iv: "9719aca15b9611b9cf6c0bcbfc04cafb"
server_key_version: "encryptionkey@0"
I20260320 02:59:08.769021 879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:08.795958 879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:08.800417 879 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:08.800930 879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:08.801628 879 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:08.802624 879 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:08.802688 879 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.802742 879 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:08.802757 879 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:08.823499 879 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:40831
I20260320 02:59:08.823550 1010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:40831 every 8 connection(s)
I20260320 02:59:08.825265 879 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:59:08.832794 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 879
I20260320 02:59:08.833060 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:59:08.833540 24081 external_mini_cluster.cc:1468] Setting key 2f205e889a5b0fb9de535cd5dd934500
Mar 20 02:59:08 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:59:08.844358 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:08.827629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:58935 (local address 127.23.132.126:39437)
0320 02:59:08.828076 (+ 447us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:08.828080 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:08.828708 (+ 628us) server_negotiation.cc:408] Connection header received
0320 02:59:08.830069 (+ 1361us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:08.830073 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:08.830127 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:08.830206 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:08.833997 (+ 3791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.834550 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:08.835983 (+ 1433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.836166 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:08.840104 (+ 3938us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:08.840126 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:08.840128 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:08.840166 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:08.842330 (+ 2164us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:08.842910 (+ 580us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:08.842913 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:08.842915 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:08.842960 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:08.843284 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:08.843287 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:08.843289 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:08.843451 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:08.843534 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:08.844061 (+ 527us) server_negotiation.cc:300] Negotiation successful
0320 02:59:08.844187 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260320 02:59:08.845089 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:08.827965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:58935)
0320 02:59:08.828499 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:08.828533 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:08.829722 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:08.830538 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:08.830551 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:08.831437 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:08.833601 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:08.833627 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.834780 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:08.834786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:08.835565 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:08.835581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:08.836129 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:08.837273 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:08.837313 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:08.839750 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:08.842487 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:08.842494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:08.842506 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:08.842803 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:08.843065 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:08.843068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:08.843070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:08.843193 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:08.843548 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:08.843555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:08.843812 (+ 257us) client_negotiation.cc:769] Sending connection context
0320 02:59:08.844009 (+ 197us) client_negotiation.cc:240] Negotiation successful
0320 02:59:08.844302 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":122,"threads_started":1}
I20260320 02:59:08.846572 1011 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39437
I20260320 02:59:08.846886 1011 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:08.847607 1011 heartbeater.cc:507] Master 127.23.132.126:39437 requested a full tablet report, sending...
I20260320 02:59:08.849644 538 ts_manager.cc:194] Registered new tserver with Master: 0637773bb2c3492e984e4fe8681330ff (127.23.132.67:40831)
I20260320 02:59:08.850525 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:59:08.850770 538 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:58935
WARNING: no policy specified for oryx/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.65.keytab.
Entry for principal oryx/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.65.keytab.
WARNING: no policy specified for oryx/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.66.keytab.
Entry for principal oryx/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.66.keytab.
WARNING: no policy specified for oryx/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.67.keytab.
Entry for principal oryx/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.67.keytab.
I20260320 02:59:08.950660 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 604
I20260320 02:59:08.957078 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 741
I20260320 02:59:08.962674 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 879
I20260320 02:59:08.968269 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 507
I20260320 02:59:08.974853 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39437
--webserver_interface=127.23.132.126
--webserver_port=37517
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:09.092705 1020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:09.092955 1020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:09.093010 1020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:09.096637 1020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:09.096845 1020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:09.097003 1020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:09.097026 1020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:09.097045 1020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:09.102680 1020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=37517
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:09.103893 1020 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:09.104703 1020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:09.110831 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:09.110791 1025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:09.111135 1020 server_base.cc:1061] running on GCE node
W20260320 02:59:09.111217 1026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:09.111686 1020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:09.112962 1020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:09.114162 1020 hybrid_clock.cc:648] HybridClock initialized: now 1773975549114127 us; error 49 us; skew 500 ppm
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975549, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:09.117635 1020 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:59:09.118844 1020 webserver.cc:492] Webserver started at http://127.23.132.126:37517/ using document root <none> and password file <none>
I20260320 02:59:09.119375 1020 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:09.119438 1020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:09.123241 1020 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260320 02:59:09.125489 1036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.126991 1020 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260320 02:59:09.127113 1020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "59abe9003d844388a7d13a33bbce392d"
format_stamp: "Formatted at 2026-03-20 02:59:07 on dist-test-slave-v3hj"
server_key: "60b38d84bedb82807ea84a95fdbff78a"
server_key_iv: "a233c21e269b62561c797e97799dacbc"
server_key_version: "encryptionkey@0"
I20260320 02:59:09.127540 1020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:09.141052 1020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:09.145531 1020 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:09.146031 1020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:09.156178 1020 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:39437
I20260320 02:59:09.156452 1089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:39437 every 8 connection(s)
I20260320 02:59:09.157332 1020 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:59:09.162111 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1020
I20260320 02:59:09.162621 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:38887
--local_ip_for_outbound_sockets=127.23.132.65
--tserver_master_addrs=127.23.132.126:39437
--webserver_port=35901
--webserver_interface=127.23.132.65
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.132.65
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260320 02:59:09.164639 1090 sys_catalog.cc:263] Verifying existing consensus state
I20260320 02:59:09.166680 1090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Bootstrap starting.
I20260320 02:59:09.188477 1090 log.cc:826] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:09.195884 1090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260320 02:59:09.196424 1090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Bootstrap complete.
I20260320 02:59:09.199975 1090 raft_consensus.cc:359] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:09.200320 1090 raft_consensus.cc:740] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59abe9003d844388a7d13a33bbce392d, State: Initialized, Role: FOLLOWER
I20260320 02:59:09.200824 1090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:09.200941 1090 raft_consensus.cc:399] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:59:09.200999 1090 raft_consensus.cc:493] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:59:09.201102 1090 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 1 FOLLOWER]: Advancing to term 2
I20260320 02:59:09.202664 1090 raft_consensus.cc:515] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:09.202980 1090 leader_election.cc:304] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59abe9003d844388a7d13a33bbce392d; no voters:
I20260320 02:59:09.203262 1090 leader_election.cc:290] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260320 02:59:09.203363 1094 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 2 FOLLOWER]: Leader election won for term 2
I20260320 02:59:09.203532 1094 raft_consensus.cc:697] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [term 2 LEADER]: Becoming Leader. State: Replica: 59abe9003d844388a7d13a33bbce392d, State: Running, Role: LEADER
I20260320 02:59:09.203806 1094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } }
I20260320 02:59:09.204411 1090 sys_catalog.cc:565] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:59:09.205189 1095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "59abe9003d844388a7d13a33bbce392d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } } }
I20260320 02:59:09.205248 1096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59abe9003d844388a7d13a33bbce392d. Latest consensus state: current_term: 2 leader_uuid: "59abe9003d844388a7d13a33bbce392d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59abe9003d844388a7d13a33bbce392d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39437 } } }
I20260320 02:59:09.205314 1095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:09.205344 1096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:09.210366 1110 catalog_manager.cc:1269] Loaded cluster ID: bea302741a434b998baf9bbeabbc14a0
I20260320 02:59:09.210438 1110 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: loading cluster ID for follower catalog manager: success
I20260320 02:59:09.212518 1110 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: acquiring CA information for follower catalog manager: success
I20260320 02:59:09.214143 1110 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260320 02:59:09.214767 1099 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:59:09.216102 1099 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:59:09.216563 1099 catalog_manager.cc:1269] Loaded cluster ID: bea302741a434b998baf9bbeabbc14a0
I20260320 02:59:09.216631 1099 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:59:09.217296 1099 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:59:09.217834 1099 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 59abe9003d844388a7d13a33bbce392d: Loaded TSK: 0
I20260320 02:59:09.218554 1099 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260320 02:59:09.289336 1092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:09.289567 1092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:09.289659 1092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:09.294247 1092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:09.294341 1092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:09.294435 1092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:59:09.300096 1092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.65.keytab
--principal=oryx/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:38887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=35901
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:09.301275 1092 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:09.302145 1092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:09.310360 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:09.310438 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:09.311627 1117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:09.311764 1092 server_base.cc:1061] running on GCE node
I20260320 02:59:09.312233 1092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:09.312917 1092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:09.316218 1092 hybrid_clock.cc:648] HybridClock initialized: now 1773975549316191 us; error 30 us; skew 500 ppm
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975549, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:09.321024 1092 init.cc:377] Logged in from keytab as oryx/127.23.132.65@KRBTEST.COM (short username oryx)
I20260320 02:59:09.322767 1092 webserver.cc:492] Webserver started at http://127.23.132.65:35901/ using document root <none> and password file <none>
I20260320 02:59:09.323473 1092 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:09.323565 1092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:09.328534 1092 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260320 02:59:09.332069 1126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.333756 1092 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:09.334115 1092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "71d7dba81b3f476cab08c05878da4918"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "6ea0b9fddbd22db50abb4307f39ff615"
server_key_iv: "865a9304962a38141d50f00e66d13022"
server_key_version: "encryptionkey@0"
I20260320 02:59:09.334560 1092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:09.348088 1092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:09.351812 1092 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:09.351991 1092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:09.352555 1092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:09.353473 1092 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:09.353525 1092 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.353577 1092 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:09.353606 1092 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.370994 1092 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:38887
I20260320 02:59:09.371006 1241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:38887 every 8 connection(s)
I20260320 02:59:09.372197 1092 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:59:09.381615 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1092
I20260320 02:59:09.382223 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:33041
--local_ip_for_outbound_sockets=127.23.132.66
--tserver_master_addrs=127.23.132.126:39437
--webserver_port=36643
--webserver_interface=127.23.132.66
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.132.66
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:09.388104 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.375561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:51181 (local address 127.23.132.126:39437)
0320 02:59:09.378362 (+ 2801us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.378375 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.378415 (+ 40us) server_negotiation.cc:408] Connection header received
0320 02:59:09.378725 (+ 310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.378738 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.379080 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.379457 (+ 377us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.381589 (+ 2132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.382971 (+ 1382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.383685 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.383971 (+ 286us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.387396 (+ 3425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:51181: BlockingRecv error: recv got EOF from 127.23.132.65:51181 (error 108)
Metrics: {"server-negotiator.queue_time_us":2595,"thread_start_us":120,"threads_started":1}
I20260320 02:59:09.388103 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.375561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:51181)
0320 02:59:09.376095 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.376136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.377500 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.379562 (+ 2062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.379574 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.380273 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.381367 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.381386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.383145 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.383149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.383562 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.383571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.383797 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.384509 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.384535 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.387163 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
W20260320 02:59:09.388886 1242 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:39437 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.132.126:39437: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:09.395931 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.389618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:47621)
0320 02:59:09.390070 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.390084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.390189 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.390763 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.390766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.391016 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.391221 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.391228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.391980 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.391984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.392350 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.392356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.392457 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.393318 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.393334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.395780 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":340}
W20260320 02:59:09.396018 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.389648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:47621 (local address 127.23.132.126:39437)
0320 02:59:09.389882 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.389886 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.390150 (+ 264us) server_negotiation.cc:408] Connection header received
0320 02:59:09.390516 (+ 366us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.390520 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.390580 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.390729 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.391358 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.391866 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.392450 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.393260 (+ 810us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.395915 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:47621: BlockingRecv error: recv got EOF from 127.23.132.65:47621 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:09.404955 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.396820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:57277)
0320 02:59:09.396993 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.397006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.397516 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.398416 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.398419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.398589 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.398891 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.398901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.400260 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.400433 (+ 173us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.400806 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.400816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.400944 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.402099 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.402116 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.404809 (+ 2693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260320 02:59:09.405122 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.396883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:57277 (local address 127.23.132.126:39437)
0320 02:59:09.396992 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.396996 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.397136 (+ 140us) server_negotiation.cc:408] Connection header received
0320 02:59:09.397949 (+ 813us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.397953 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.398097 (+ 144us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.398278 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.399007 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.400137 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.400950 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.402083 (+ 1133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.405010 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:57277: BlockingRecv error: recv got EOF from 127.23.132.65:57277 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260320 02:59:09.405674 1242 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260320 02:59:09.545054 1246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:09.545296 1246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:09.545358 1246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:09.550386 1246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:09.550472 1246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:09.550555 1246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:59:09.555774 1246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.66.keytab
--principal=oryx/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:33041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=36643
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:09.556854 1246 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:09.558085 1246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:09.567569 1251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:09.567569 1252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:09.567978 1254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:09.568565 1246 server_base.cc:1061] running on GCE node
I20260320 02:59:09.569420 1246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:09.570372 1246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:09.573706 1246 hybrid_clock.cc:648] HybridClock initialized: now 1773975549573792 us; error 297 us; skew 500 ppm
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975549, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:09.578027 1246 init.cc:377] Logged in from keytab as oryx/127.23.132.66@KRBTEST.COM (short username oryx)
I20260320 02:59:09.579203 1246 webserver.cc:492] Webserver started at http://127.23.132.66:36643/ using document root <none> and password file <none>
I20260320 02:59:09.580058 1246 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:09.580233 1246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:09.584558 1246 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260320 02:59:09.586906 1262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.588795 1246 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:09.589057 1246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "67e74fe9385e4978b3a23955a907d6e5"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "10b469e15262d1e0ab73eb133fcace67"
server_key_iv: "41614e773f23c3b857600efa54208382"
server_key_version: "encryptionkey@0"
I20260320 02:59:09.589577 1246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:09.651834 1246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:09.655210 1246 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:09.655402 1246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:09.655974 1246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:09.656857 1246 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:09.656904 1246 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.656955 1246 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:09.656983 1246 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.676430 1246 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:33041
I20260320 02:59:09.676462 1377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:33041 every 8 connection(s)
I20260320 02:59:09.677489 1246 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:59:09.682789 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1246
I20260320 02:59:09.683195 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:40831
--local_ip_for_outbound_sockets=127.23.132.67
--tserver_master_addrs=127.23.132.126:39437
--webserver_port=41135
--webserver_interface=127.23.132.67
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.132.67
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:09.694584 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.680962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:40211 (local address 127.23.132.126:39437)
0320 02:59:09.681733 (+ 771us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.681738 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.682465 (+ 727us) server_negotiation.cc:408] Connection header received
0320 02:59:09.684780 (+ 2315us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.684785 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.684917 (+ 132us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.684999 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.687420 (+ 2421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.688106 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.688863 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.689031 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.694443 (+ 5412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:40211: BlockingRecv error: recv got EOF from 127.23.132.66:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":634}
I20260320 02:59:09.695102 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.681395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:40211)
0320 02:59:09.682283 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.682321 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.683144 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.685215 (+ 2071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.685225 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.685648 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.687226 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.687245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.688339 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.688342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.688747 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.688756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.688987 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.690466 (+ 1479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.690497 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.694273 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":394,"threads_started":1}
W20260320 02:59:09.696252 1378 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:39437 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.132.126:39437: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:09.706355 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.696818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:39757)
0320 02:59:09.696946 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.696962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.697056 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.697532 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.697536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.697733 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.697992 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.698001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.699006 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.699009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.699405 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.699412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.699505 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.700043 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.700129 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.706206 (+ 6077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260320 02:59:09.707315 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.697008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:39757 (local address 127.23.132.126:39437)
0320 02:59:09.697145 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.697150 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.697166 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:09.697215 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.697218 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.697271 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.697370 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.698118 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.698868 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.699514 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.699865 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.707147 (+ 7282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:39757: BlockingRecv error: recv got EOF from 127.23.132.66:39757 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:09.717716 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.707682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:51199)
0320 02:59:09.708041 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.708060 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.708184 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.709579 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.709582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.709756 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.710195 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.710213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.712178 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.712182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.712996 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.713009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.713173 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.714450 (+ 1277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.714467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.717572 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143}
W20260320 02:59:09.717963 1245 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.707888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:51199 (local address 127.23.132.126:39437)
0320 02:59:09.708163 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.708168 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.708184 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:09.709402 (+ 1218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.709406 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.709461 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.709539 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.711060 (+ 1521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.712002 (+ 942us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.713216 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.713424 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.717850 (+ 4426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:51199: BlockingRecv error: recv got EOF from 127.23.132.66:51199 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
W20260320 02:59:09.718221 1378 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260320 02:59:09.816506 1381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:09.816824 1381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:09.816897 1381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:09.821168 1381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:09.821260 1381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:09.821359 1381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:59:09.830174 1381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/krb5kdc/oryx_127.23.132.67.keytab
--principal=oryx/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:40831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=41135
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:39437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:09.831375 1381 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:09.832289 1381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:09.840296 1386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:09.840304 1387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:09.841459 1381 server_base.cc:1061] running on GCE node
W20260320 02:59:09.841504 1389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:09.842516 1381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:09.843196 1381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:09.844403 1381 hybrid_clock.cc:648] HybridClock initialized: now 1773975549844349 us; error 69 us; skew 500 ppm
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975549, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:09.848630 1381 init.cc:377] Logged in from keytab as oryx/127.23.132.67@KRBTEST.COM (short username oryx)
I20260320 02:59:09.850343 1381 webserver.cc:492] Webserver started at http://127.23.132.67:41135/ using document root <none> and password file <none>
I20260320 02:59:09.851161 1381 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:09.851254 1381 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:09.857498 1381 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:09.859679 1396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.861387 1381 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:09.861526 1381 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "0637773bb2c3492e984e4fe8681330ff"
format_stamp: "Formatted at 2026-03-20 02:59:08 on dist-test-slave-v3hj"
server_key: "050a74a2b0712593f47976fff7b96f2a"
server_key_iv: "9719aca15b9611b9cf6c0bcbfc04cafb"
server_key_version: "encryptionkey@0"
I20260320 02:59:09.862043 1381 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:09.886564 1381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:09.890599 1381 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:09.890862 1381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:09.891494 1381 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:09.892405 1381 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:09.892454 1381 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.892506 1381 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:09.892534 1381 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:09.909502 1381 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:40831
I20260320 02:59:09.909534 1512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:40831 every 8 connection(s)
I20260320 02:59:09.910758 1381 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:59:09.911929 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1381
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:09.923099 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.912905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.23.132.126:39437)
0320 02:59:09.913067 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.913071 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.913584 (+ 513us) server_negotiation.cc:408] Connection header received
0320 02:59:09.913887 (+ 303us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.913892 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.914009 (+ 117us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.914105 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.915027 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.915689 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.916413 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.916594 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.918349 (+ 1755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:09.918370 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:09.918381 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:09.918421 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:09.920690 (+ 2269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:09.921163 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:09.921168 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:09.921172 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:09.921248 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:09.921458 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:09.921461 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:09.921463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:09.921976 (+ 513us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:09.922164 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:09.922591 (+ 427us) server_negotiation.cc:300] Negotiation successful
0320 02:59:09.922942 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:09.924929 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.913629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:40171 (local address 127.23.132.126:39437)
0320 02:59:09.914034 (+ 405us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.914038 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.914436 (+ 398us) server_negotiation.cc:408] Connection header received
0320 02:59:09.915916 (+ 1480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.915920 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.915970 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.916042 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.918892 (+ 2850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.919743 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.920662 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.920997 (+ 335us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.924795 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:40171: BlockingRecv error: recv got EOF from 127.23.132.67:40171 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":73,"threads_started":1}
I20260320 02:59:09.925539 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.913676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:40171)
0320 02:59:09.914316 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.914356 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.915601 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.916280 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.916289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.916862 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.918647 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.918669 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.920068 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.920072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.920538 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.920547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.920722 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.921440 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.921464 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.924662 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":156,"threads_started":1}
W20260320 02:59:09.926311 1513 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:39437 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.132.126:39437: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:09.932668 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.927022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:40381)
0320 02:59:09.927202 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.927214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.927295 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.927739 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.927744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.927995 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.928319 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.928328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.929599 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.929603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.930186 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.930197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.930315 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.930810 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.930823 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.932558 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260320 02:59:09.933193 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.927104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:40381 (local address 127.23.132.126:39437)
0320 02:59:09.927215 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.927218 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.927253 (+ 35us) server_negotiation.cc:408] Connection header received
0320 02:59:09.927426 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.927429 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.927475 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.927549 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.928528 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.929429 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.930336 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.930491 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.932960 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:40381: BlockingRecv error: recv got EOF from 127.23.132.67:40381 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:09 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:09.939213 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:09.933358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:39091 (local address 127.23.132.126:39437)
0320 02:59:09.933977 (+ 619us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:09.933981 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:09.933993 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:59:09.934025 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:09.934028 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:09.934067 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:09.934132 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:09.934704 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.935486 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.936389 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.936596 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.939106 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:39091: BlockingRecv error: recv got EOF from 127.23.132.67:39091 (error 108)
Metrics: {"server-negotiator.queue_time_us":380}
I20260320 02:59:09.939273 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:09.933281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:39091)
0320 02:59:09.933495 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:09.933507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:09.933570 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:09.934198 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:09.934201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:09.934346 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:09.934514 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.934519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.935634 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:09.935637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:09.936065 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:09.936074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:09.936174 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:09.936804 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:09.936816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:09.939136 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147}
W20260320 02:59:09.939581 1513 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Mar 20 02:59:10 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:10 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:10.413359 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:10.406414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:51343)
0320 02:59:10.406926 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:10.406944 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:10.407062 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:10.407500 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:10.407504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:10.407915 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:10.408147 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:10.408155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.409110 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:10.409114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:10.409578 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:10.409587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.409743 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:10.410493 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:10.410508 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:10.413205 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":85,"threads_started":1}
W20260320 02:59:10.413451 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:10.406521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:51343 (local address 127.23.132.126:39437)
0320 02:59:10.406675 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:10.406679 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:10.407035 (+ 356us) server_negotiation.cc:408] Connection header received
0320 02:59:10.407199 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:10.407202 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:10.407336 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:10.407440 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:10.408282 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.408910 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:10.409704 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.410006 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:10.413354 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:51343: BlockingRecv error: recv got EOF from 127.23.132.65:51343 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Mar 20 02:59:10 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:10 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:10.726119 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:10.719242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:53555)
0320 02:59:10.719604 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:10.719620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:10.719745 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:10.720168 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:10.720171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:10.720447 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:10.720697 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:10.720704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.721527 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:10.721530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:10.722229 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:10.722245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.722393 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:10.723259 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:10.723274 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:10.725975 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
W20260320 02:59:10.726219 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:10.719382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:53555 (local address 127.23.132.126:39437)
0320 02:59:10.719582 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:10.719586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:10.719760 (+ 174us) server_negotiation.cc:408] Connection header received
0320 02:59:10.719829 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:10.719831 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:10.719948 (+ 117us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:10.720021 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:10.720839 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.721403 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:10.722477 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.722733 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:10.726107 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:53555: BlockingRecv error: recv got EOF from 127.23.132.66:53555 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Mar 20 02:59:10 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:10 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:10.947322 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:10.940335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:58757)
0320 02:59:10.940606 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:10.940620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:10.940863 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:10.941243 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:10.941246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:10.941527 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:10.941741 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:10.941748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.942964 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:10.942977 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:10.943505 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:10.943515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.943644 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:10.944615 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:10.944630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:10.947183 (+ 2553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
W20260320 02:59:10.947526 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:10.940660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:58757 (local address 127.23.132.126:39437)
0320 02:59:10.940800 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:10.940804 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:10.940820 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:10.941028 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:10.941032 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:10.941082 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:10.941159 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:10.941963 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.942643 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:10.943697 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:10.943897 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:10.947411 (+ 3514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:58757: BlockingRecv error: recv got EOF from 127.23.132.67:58757 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Mar 20 02:59:11 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:11 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:11.422261 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:11.414590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:39009)
0320 02:59:11.415141 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:11.415168 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:11.415410 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:11.415683 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:11.415687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:11.415927 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:11.416124 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:11.416132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.418301 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:11.418307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:11.418770 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:11.418781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.418962 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:11.419956 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:11.419993 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:11.422137 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":96,"threads_started":1}
W20260320 02:59:11.422483 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:11.414748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:39009 (local address 127.23.132.126:39437)
0320 02:59:11.415112 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:11.415171 (+ 59us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:11.415370 (+ 199us) server_negotiation.cc:408] Connection header received
0320 02:59:11.415517 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:11.415521 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:11.415574 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:11.415719 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:11.416409 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.417886 (+ 1477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:11.419078 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.419461 (+ 383us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:11.422352 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:39009: BlockingRecv error: recv got EOF from 127.23.132.65:39009 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Mar 20 02:59:11 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:11 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:11.735881 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:11.727125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:53637)
0320 02:59:11.727476 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:11.727492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:11.727598 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:11.727853 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:11.727856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:11.728192 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:11.728411 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:11.728418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.729165 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:11.729220 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:11.729627 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:11.729637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.729827 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:11.731124 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:11.731277 (+ 153us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:11.735743 (+ 4466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":84,"threads_started":1}
W20260320 02:59:11.736305 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:11.727406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:53637 (local address 127.23.132.126:39437)
0320 02:59:11.727574 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:11.727578 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:11.727594 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:11.727706 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:11.727709 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:11.727754 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:11.727821 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:11.728554 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.729057 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:11.729814 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.730295 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:11.736167 (+ 5872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:53637: BlockingRecv error: recv got EOF from 127.23.132.66:53637 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Mar 20 02:59:11 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:11 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:11.956820 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:11.948250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:53573)
0320 02:59:11.948526 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:11.948540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:11.948683 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:11.949074 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:11.949077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:11.949252 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:11.949441 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:11.949447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.951239 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:11.951243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:11.951665 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:11.951674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.951803 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:11.952346 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:11.952359 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:11.956662 (+ 4303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
W20260320 02:59:11.957566 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:11.948461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:53573 (local address 127.23.132.126:39437)
0320 02:59:11.948615 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:11.948619 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:11.948633 (+ 14us) server_negotiation.cc:408] Connection header received
0320 02:59:11.948833 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:11.948836 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:11.948886 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:11.948994 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:11.949592 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.951102 (+ 1510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:11.952598 (+ 1496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:11.953041 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:11.957475 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:53573: BlockingRecv error: recv got EOF from 127.23.132.67:53573 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Mar 20 02:59:12 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:12 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:12.435169 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:12.423220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:47395)
0320 02:59:12.423570 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:12.423584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:12.423679 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:12.423995 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:12.423998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:12.424175 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:12.424379 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:12.424384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.425212 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:12.425216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:12.425611 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:12.425619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.425712 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:12.431387 (+ 5675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:12.431405 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:12.435009 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":148,"threads_started":1}
W20260320 02:59:12.435684 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:12.423452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:47395 (local address 127.23.132.126:39437)
0320 02:59:12.423696 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:12.423700 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:12.423750 (+ 50us) server_negotiation.cc:408] Connection header received
0320 02:59:12.423797 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:12.423801 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:12.423847 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:12.423912 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:12.424509 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.425075 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:12.425746 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.426093 (+ 347us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:12.435558 (+ 9465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:47395: BlockingRecv error: recv got EOF from 127.23.132.65:47395 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Mar 20 02:59:12 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:12 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:12.744699 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:12.736962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:39847)
0320 02:59:12.737482 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:12.737497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:12.737664 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:12.738758 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:12.738761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:12.739048 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:12.739245 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:12.739251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.740613 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:12.740617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:12.741485 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:12.741495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.741617 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:12.742327 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:12.742343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:12.744482 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":87,"threads_started":1}
W20260320 02:59:12.744890 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:12.737418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:39847 (local address 127.23.132.126:39437)
0320 02:59:12.737545 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:12.737548 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:12.737564 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:12.737866 (+ 302us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:12.737870 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:12.738056 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:12.738617 (+ 561us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:12.739533 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.740462 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:12.741624 (+ 1162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.741843 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:12.744709 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:39847: BlockingRecv error: recv got EOF from 127.23.132.66:39847 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Mar 20 02:59:12 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:12 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:12.964748 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:12.957995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:54257)
0320 02:59:12.958273 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:12.958286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:12.958398 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:12.958799 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:12.958803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:12.959596 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:12.960236 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:12.960245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.961523 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:12.961526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:12.962014 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:12.962025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.962132 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:12.962657 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:12.962670 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:12.964556 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
W20260320 02:59:12.964920 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:12.958136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:54257 (local address 127.23.132.126:39437)
0320 02:59:12.958282 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:12.958287 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:12.958369 (+ 82us) server_negotiation.cc:408] Connection header received
0320 02:59:12.958496 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:12.958500 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:12.958549 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:12.958627 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:12.960599 (+ 1972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.961322 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:12.962157 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:12.962298 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:12.964793 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:54257: BlockingRecv error: recv got EOF from 127.23.132.67:54257 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Mar 20 02:59:13 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:13 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:13.443017 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:13.436717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:50605 (local address 127.23.132.126:39437)
0320 02:59:13.436877 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:13.436880 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:13.436896 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:13.437058 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:13.437062 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:13.437111 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:13.437195 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:13.437985 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.438500 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:13.439739 (+ 1239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.439907 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:13.442916 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:50605: BlockingRecv error: recv got EOF from 127.23.132.65:50605 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260320 02:59:13.443041 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:13.436538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:50605)
0320 02:59:13.436804 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:13.436817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:13.436919 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:13.437297 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:13.437300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:13.437476 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:13.437673 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:13.437680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.438777 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:13.438781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:13.439246 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:13.439255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.439372 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:13.440492 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:13.440514 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:13.442793 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
Mar 20 02:59:13 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:13 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:13.753487 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:13.745774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:50469)
0320 02:59:13.746325 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:13.746361 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:13.746646 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:13.747072 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:13.747075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:13.747404 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:13.747627 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:13.747634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.748465 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:13.748468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:13.748838 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:13.748845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.748937 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:13.749598 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:13.749620 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:13.753312 (+ 3692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":94,"threads_started":1}
W20260320 02:59:13.753556 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:13.745890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:50469 (local address 127.23.132.126:39437)
0320 02:59:13.746051 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:13.746056 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:13.746546 (+ 490us) server_negotiation.cc:408] Connection header received
0320 02:59:13.746852 (+ 306us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:13.746855 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:13.746916 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:13.747004 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:13.747836 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.748328 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:13.749005 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.749163 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:13.753455 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:50469: BlockingRecv error: recv got EOF from 127.23.132.66:50469 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Mar 20 02:59:13 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:13 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:13.971187 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:13.965653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:58165)
0320 02:59:13.965970 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:13.965985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:13.966099 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:13.966506 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:13.966509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:13.966717 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:13.966915 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:13.966922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.967667 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:13.967670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:13.968087 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:13.968094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.968199 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:13.968821 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:13.968834 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:13.971066 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
W20260320 02:59:13.971738 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:13.965824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:58165 (local address 127.23.132.126:39437)
0320 02:59:13.965977 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:13.965981 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:13.966052 (+ 71us) server_negotiation.cc:408] Connection header received
0320 02:59:13.966179 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:13.966182 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:13.966324 (+ 142us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:13.966409 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:13.967039 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.967562 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:13.968196 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:13.968363 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:13.971594 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:58165: BlockingRecv error: recv got EOF from 127.23.132.67:58165 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Mar 20 02:59:14 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:14 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:14.450240 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:14.444028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:45477)
0320 02:59:14.444349 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:14.444364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:14.444480 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:14.444955 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:14.444959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:14.445222 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:14.445453 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:14.445461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.446258 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:14.446262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:14.446824 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:14.446836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.447062 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:14.447610 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:14.447624 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:14.450122 (+ 2498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
W20260320 02:59:14.450640 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:14.444179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:45477 (local address 127.23.132.126:39437)
0320 02:59:14.444352 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:14.444357 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:14.444439 (+ 82us) server_negotiation.cc:408] Connection header received
0320 02:59:14.444573 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:14.444577 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:14.444746 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:14.444841 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:14.445570 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.446150 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:14.447046 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.447220 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:14.450315 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:45477: BlockingRecv error: recv got EOF from 127.23.132.65:45477 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Mar 20 02:59:14 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:14 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:14.762451 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:14.754918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:44107)
0320 02:59:14.755214 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:14.755227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:14.755314 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:14.755886 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:14.755890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:14.756102 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:14.756292 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:14.756298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.757001 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:14.757005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:14.757369 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:14.757376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.757476 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:14.758687 (+ 1211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:14.758704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:14.762324 (+ 3620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
W20260320 02:59:14.762630 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:14.755468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:44107 (local address 127.23.132.126:39437)
0320 02:59:14.755637 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:14.755642 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:14.755658 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:14.755704 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:14.755707 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:14.755752 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:14.755827 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:14.756411 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.756897 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:14.757473 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.757644 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:14.762499 (+ 4855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:44107: BlockingRecv error: recv got EOF from 127.23.132.66:44107 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Mar 20 02:59:14 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:14 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:14.982291 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:14.972248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:36655)
0320 02:59:14.972735 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:14.972757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:14.972882 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:14.973587 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:14.973590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:14.973926 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:14.974712 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:14.974955 (+ 243us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.976264 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:14.976268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:14.976726 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:14.976737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.978671 (+ 1934us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:14.979320 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:14.979334 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:14.982167 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":74,"threads_started":1}
W20260320 02:59:14.982513 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:14.972725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:36655 (local address 127.23.132.126:39437)
0320 02:59:14.973021 (+ 296us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:14.973035 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:14.973249 (+ 214us) server_negotiation.cc:408] Connection header received
0320 02:59:14.973295 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:14.973298 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:14.973343 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:14.973573 (+ 230us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:14.975319 (+ 1746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.976118 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:14.978755 (+ 2637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:14.979009 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:14.982396 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:36655: BlockingRecv error: recv got EOF from 127.23.132.67:36655 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Mar 20 02:59:15 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:15 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:15.459945 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:15.451237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:36299)
0320 02:59:15.451716 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:15.451751 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:15.452064 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:15.452430 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:15.452433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:15.452609 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:15.452830 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:15.452836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.454738 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:15.454742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:15.455667 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:15.455679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.456076 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:15.456675 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:15.456691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:15.459787 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
W20260320 02:59:15.460335 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:15.451378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:36299 (local address 127.23.132.126:39437)
0320 02:59:15.451524 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:15.451528 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:15.451991 (+ 463us) server_negotiation.cc:408] Connection header received
0320 02:59:15.452222 (+ 231us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:15.452225 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:15.452274 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:15.452363 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:15.453043 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.453537 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:15.456190 (+ 2653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.456452 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:15.459979 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:36299: BlockingRecv error: recv got EOF from 127.23.132.65:36299 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Mar 20 02:59:15 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:15 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:15.769421 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:15.763210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:52585)
0320 02:59:15.763739 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:15.763763 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:15.763884 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:15.764305 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:15.764308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:15.764499 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:15.764686 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:15.764692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.765738 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:15.765743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:15.766266 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:15.766276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.766403 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:15.766985 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:15.766997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:15.769265 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":111,"threads_started":1}
W20260320 02:59:15.769515 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:15.763276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:52585 (local address 127.23.132.126:39437)
0320 02:59:15.763403 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:15.763407 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:15.763856 (+ 449us) server_negotiation.cc:408] Connection header received
0320 02:59:15.763971 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:15.763975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:15.764165 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:15.764260 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:15.764812 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.765458 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:15.766396 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.766590 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:15.769412 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:52585: BlockingRecv error: recv got EOF from 127.23.132.66:52585 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Mar 20 02:59:15 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:15 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:15.989071 1535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:15.983251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:53653)
0320 02:59:15.983573 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:15.983587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:15.983701 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:15.984019 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:15.984022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:15.984198 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:15.984390 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:15.984396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.985477 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:15.985482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:15.985970 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:15.985982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.986124 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:15.986878 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:15.986892 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:15.988914 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":138,"threads_started":1}
W20260320 02:59:15.989173 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:15.983290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:53653 (local address 127.23.132.126:39437)
0320 02:59:15.983417 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:15.983420 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:15.983755 (+ 335us) server_negotiation.cc:408] Connection header received
0320 02:59:15.983799 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:15.983801 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:15.983847 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:15.983917 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:15.984599 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.985260 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:15.986161 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:15.986451 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:15.989070 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:53653: BlockingRecv error: recv got EOF from 127.23.132.67:53653 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Mar 20 02:59:16 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:16 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:16.466305 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:16.461311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:44607)
0320 02:59:16.461642 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:16.461658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:16.461776 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:16.462098 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:16.462100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:16.462288 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:16.462485 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:16.462490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.463291 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:16.463294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:16.463693 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:16.463701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.463800 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:16.464433 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:16.464446 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:16.466182 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
W20260320 02:59:16.466523 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:16.461464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:44607 (local address 127.23.132.126:39437)
0320 02:59:16.461598 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:16.461601 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:16.461847 (+ 246us) server_negotiation.cc:408] Connection header received
0320 02:59:16.461898 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:16.461901 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:16.461949 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:16.462039 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:16.462628 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.463155 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:16.463852 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.464034 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:16.466404 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:44607: BlockingRecv error: recv got EOF from 127.23.132.65:44607 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Mar 20 02:59:16 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:16 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:16.778031 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:16.770377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:39185)
0320 02:59:16.770685 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:16.770698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:16.770811 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:16.771208 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:16.771211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:16.771404 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:16.771805 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:16.771814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.772697 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:16.772702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:16.773422 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:16.773440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.773557 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:16.774712 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:16.774735 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:16.777719 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
W20260320 02:59:16.778200 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:16.770527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:39185 (local address 127.23.132.126:39437)
0320 02:59:16.770669 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:16.770673 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:16.770781 (+ 108us) server_negotiation.cc:408] Connection header received
0320 02:59:16.770992 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:16.770995 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:16.771044 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:16.771141 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:16.772000 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.772493 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:16.773588 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.773944 (+ 356us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:16.778097 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:39185: BlockingRecv error: recv got EOF from 127.23.132.66:39185 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Mar 20 02:59:16 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:16 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:16.996930 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:16.990051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:43447)
0320 02:59:16.990341 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:16.990355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:16.990447 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:16.991107 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:16.991110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:16.991291 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:16.991491 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:16.991497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.992827 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:16.992831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:16.993450 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:16.993459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.993577 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:16.994129 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:16.994142 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:16.996720 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
W20260320 02:59:16.997126 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:16.990164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:43447 (local address 127.23.132.126:39437)
0320 02:59:16.990417 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:16.990423 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:16.990453 (+ 30us) server_negotiation.cc:408] Connection header received
0320 02:59:16.990560 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:16.990563 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:16.990634 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:16.991065 (+ 431us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:16.991651 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.992473 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:16.993637 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:16.993994 (+ 357us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:16.996897 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:43447: BlockingRecv error: recv got EOF from 127.23.132.67:43447 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Mar 20 02:59:17 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:17 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:17.480590 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:17.467464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:33045)
0320 02:59:17.467743 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:17.467756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:17.467857 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:17.468181 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:17.468184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:17.468357 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:17.468560 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:17.468565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.469336 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:17.469339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:17.469735 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:17.469742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.469951 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:17.477181 (+ 7230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:17.477205 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:17.480448 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
W20260320 02:59:17.480952 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:17.467632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:33045 (local address 127.23.132.126:39437)
0320 02:59:17.467765 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:17.467769 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:17.467864 (+ 95us) server_negotiation.cc:408] Connection header received
0320 02:59:17.467921 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:17.467923 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:17.468027 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:17.468123 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:17.468704 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.469211 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:17.469913 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.470108 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:17.480783 (+ 10675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:33045: BlockingRecv error: recv got EOF from 127.23.132.65:33045 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Mar 20 02:59:17 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:17 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:17.786489 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:17.780067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:38447 (local address 127.23.132.126:39437)
0320 02:59:17.780337 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:17.780342 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:17.780371 (+ 29us) server_negotiation.cc:408] Connection header received
0320 02:59:17.780551 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:17.780555 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:17.780622 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:17.780773 (+ 151us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:17.781504 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.782252 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:17.783027 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.783208 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:17.786364 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:38447: BlockingRecv error: recv got EOF from 127.23.132.66:38447 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
I20260320 02:59:17.786545 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:17.779608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:38447)
0320 02:59:17.780298 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:17.780318 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:17.780423 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:17.780891 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:17.780894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:17.781077 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:17.781311 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:17.781317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.782408 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:17.782412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:17.782801 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:17.782808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:17.782910 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:17.783643 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:17.783656 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:17.786415 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":360,"threads_started":1}
Mar 20 02:59:17 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:17 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:18.004673 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:17.998241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:42497)
0320 02:59:17.999147 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:17.999162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:17.999283 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:17.999525 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:17.999528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:17.999770 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:17.999978 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:17.999985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.001169 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:18.001174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:18.001595 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:18.001603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.001723 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:18.002536 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:18.002550 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:18.004506 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":511,"threads_started":1}
W20260320 02:59:18.004840 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:17.998307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:42497 (local address 127.23.132.126:39437)
0320 02:59:17.998445 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:17.998449 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:17.999262 (+ 813us) server_negotiation.cc:408] Connection header received
0320 02:59:17.999375 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:17.999379 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:17.999427 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:17.999492 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:18.000286 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.000902 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:18.001727 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.001973 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:18.004729 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:42497: BlockingRecv error: recv got EOF from 127.23.132.67:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Mar 20 02:59:18 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:18 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:18.486869 1543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:18.481450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:38789)
0320 02:59:18.481875 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:18.481899 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:18.482024 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:18.482275 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:18.482278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:18.482478 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:18.482673 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:18.482679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.483386 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:18.483390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:18.483763 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:18.483770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.483873 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:18.484436 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:18.484451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:18.486719 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
W20260320 02:59:18.486960 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:18.481672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:38789 (local address 127.23.132.126:39437)
0320 02:59:18.481879 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:18.481884 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:18.481973 (+ 89us) server_negotiation.cc:408] Connection header received
0320 02:59:18.482118 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:18.482122 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:18.482167 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:18.482236 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:18.482796 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.483278 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:18.483866 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.484030 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:18.486855 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:38789: BlockingRecv error: recv got EOF from 127.23.132.65:38789 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Mar 20 02:59:18 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:18 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:18.793676 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:18.787597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:41893)
0320 02:59:18.787938 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:18.787952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:18.788057 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:18.788589 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:18.788594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:18.788782 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:18.788986 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:18.788992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.789735 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:18.789738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:18.790166 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:18.790175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.790277 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:18.790823 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:18.790836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:18.793488 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":156,"threads_started":1}
W20260320 02:59:18.793941 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:18.787719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:41893 (local address 127.23.132.126:39437)
0320 02:59:18.787847 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:18.787851 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:18.788044 (+ 193us) server_negotiation.cc:408] Connection header received
0320 02:59:18.788220 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:18.788223 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:18.788269 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:18.788341 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:18.789117 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.789606 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:18.790289 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:18.790428 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:18.793754 (+ 3326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:41893: BlockingRecv error: recv got EOF from 127.23.132.66:41893 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Mar 20 02:59:19 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:19 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:19.011207 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:19.005490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:50439)
0320 02:59:19.005781 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:19.005820 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:19.006029 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:19.006397 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:19.006400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:19.006577 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:19.006816 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:19.006822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.007738 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:19.007742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:19.008124 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:19.008132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.008230 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:19.008877 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:19.008891 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:19.011080 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
W20260320 02:59:19.011868 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:19.005983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:50439 (local address 127.23.132.126:39437)
0320 02:59:19.006127 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:19.006130 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:19.006142 (+ 12us) server_negotiation.cc:408] Connection header received
0320 02:59:19.006173 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:19.006175 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:19.006217 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:19.006285 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:19.006947 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.007518 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:19.008258 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.008398 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:19.011691 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:50439: BlockingRecv error: recv got EOF from 127.23.132.67:50439 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Mar 20 02:59:19 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:19 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:19.497236 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:19.488080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:38565)
0320 02:59:19.488405 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:19.488424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:19.488559 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:19.489082 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:19.489085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:19.489406 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:19.489604 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:19.489611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.490450 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:19.490457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:19.490957 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:19.490966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.491093 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:19.493344 (+ 2251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:19.493367 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:19.497106 (+ 3739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
W20260320 02:59:19.497466 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:19.488121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:38565 (local address 127.23.132.126:39437)
0320 02:59:19.488275 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:19.488280 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:19.488522 (+ 242us) server_negotiation.cc:408] Connection header received
0320 02:59:19.488876 (+ 354us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:19.488880 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:19.488933 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:19.489001 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:19.489779 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.490328 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:19.491087 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.491273 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:19.497344 (+ 6071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:38565: BlockingRecv error: recv got EOF from 127.23.132.65:38565 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Mar 20 02:59:19 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:19 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:19.802636 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:19.794559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:40619)
0320 02:59:19.794927 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:19.794945 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:19.795140 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:19.796005 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:19.796009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:19.796499 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:19.796768 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:19.796776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.798145 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:19.798149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:19.798669 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:19.798680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.798858 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:19.800105 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:19.800148 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:19.802509 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
W20260320 02:59:19.802812 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:19.794778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:40619 (local address 127.23.132.126:39437)
0320 02:59:19.794976 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:19.794981 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:19.795120 (+ 139us) server_negotiation.cc:408] Connection header received
0320 02:59:19.795587 (+ 467us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:19.795592 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:19.795662 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:19.795753 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:19.796991 (+ 1238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.797938 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:19.798816 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:19.799016 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:19.802684 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:40619: BlockingRecv error: recv got EOF from 127.23.132.66:40619 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Mar 20 02:59:20 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:20 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:20.026139 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:20.012378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:59589)
0320 02:59:20.012720 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:20.012820 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:20.013030 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:20.014074 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:20.014077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:20.014723 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:20.015505 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:20.015533 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.017992 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:20.018009 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:20.018653 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:20.018663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.018831 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:20.021056 (+ 2225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:20.021087 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:20.025908 (+ 4821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
W20260320 02:59:20.026324 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:20.012937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:59589 (local address 127.23.132.126:39437)
0320 02:59:20.013221 (+ 284us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:20.013224 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:20.013269 (+ 45us) server_negotiation.cc:408] Connection header received
0320 02:59:20.013351 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:20.013366 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:20.013495 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:20.013754 (+ 259us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:20.016162 (+ 2408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.017644 (+ 1482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:20.018980 (+ 1336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.019324 (+ 344us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:20.026141 (+ 6817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:59589: BlockingRecv error: recv got EOF from 127.23.132.67:59589 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Mar 20 02:59:20 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:20 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:20.511520 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:20.498749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:40167)
0320 02:59:20.499665 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:20.499707 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:20.500026 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:20.500457 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:20.500460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:20.500706 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:20.501350 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:20.501382 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.503830 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:20.503836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:20.504386 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:20.504433 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.504733 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:20.506486 (+ 1753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:20.506561 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:20.511147 (+ 4586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":138,"threads_started":1}
W20260320 02:59:20.512240 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:20.498969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:40167 (local address 127.23.132.126:39437)
0320 02:59:20.499310 (+ 341us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:20.499314 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:20.499913 (+ 599us) server_negotiation.cc:408] Connection header received
0320 02:59:20.500182 (+ 269us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:20.500185 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:20.500236 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:20.500310 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:20.501872 (+ 1562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.503583 (+ 1711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:20.504702 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.505187 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:20.511734 (+ 6547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:40167: BlockingRecv error: recv got EOF from 127.23.132.65:40167 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Mar 20 02:59:20 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:20 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:20.817451 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:20.804468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:51855)
0320 02:59:20.805259 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:20.805352 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:20.805772 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:20.806888 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:20.806892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:20.807416 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:20.808030 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:20.808053 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.810009 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:20.810013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:20.810656 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:20.810665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.811151 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:20.812596 (+ 1445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:20.812618 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:20.817165 (+ 4547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":191,"threads_started":1}
W20260320 02:59:20.817647 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:20.804599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:51855 (local address 127.23.132.126:39437)
0320 02:59:20.805073 (+ 474us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:20.805106 (+ 33us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:20.805704 (+ 598us) server_negotiation.cc:408] Connection header received
0320 02:59:20.806253 (+ 549us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:20.806281 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:20.806454 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:20.806713 (+ 259us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:20.808365 (+ 1652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.809710 (+ 1345us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:20.811076 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:20.811355 (+ 279us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:20.817294 (+ 5939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:51855: BlockingRecv error: recv got EOF from 127.23.132.66:51855 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Mar 20 02:59:21 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:21 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:21.038168 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:21.027510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:57105)
0320 02:59:21.028013 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:21.028031 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:21.028168 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:21.028686 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:21.028689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:21.029029 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:21.029313 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:21.029320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.030504 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:21.030508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:21.031101 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:21.031111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.031231 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:21.032335 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:21.032422 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:21.037692 (+ 5270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":294,"threads_started":1}
W20260320 02:59:21.038295 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:21.027646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:57105 (local address 127.23.132.126:39437)
0320 02:59:21.027793 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:21.027797 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:21.028221 (+ 424us) server_negotiation.cc:408] Connection header received
0320 02:59:21.028293 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:21.028296 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:21.028374 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:21.028536 (+ 162us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:21.029525 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.030349 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:21.031331 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.031515 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:21.038172 (+ 6657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:57105: BlockingRecv error: recv got EOF from 127.23.132.67:57105 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Mar 20 02:59:21 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:21 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:21.519553 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:21.513091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:39949)
0320 02:59:21.513547 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:21.513561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:21.513706 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:21.514206 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:21.514209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:21.514511 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:21.514740 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:21.514746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.515645 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:21.515648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:21.516180 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:21.516190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.516462 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:21.517210 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:21.517224 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:21.519398 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
W20260320 02:59:21.519637 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:21.513272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:39949 (local address 127.23.132.126:39437)
0320 02:59:21.513462 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:21.513466 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:21.513710 (+ 244us) server_negotiation.cc:408] Connection header received
0320 02:59:21.513857 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:21.513861 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:21.514002 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:21.514103 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:21.514918 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.515489 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:21.516505 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.516714 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:21.519532 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:39949: BlockingRecv error: recv got EOF from 127.23.132.65:39949 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Mar 20 02:59:21 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:21 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:21.826172 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:21.819143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:56223)
0320 02:59:21.819568 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:21.819587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:21.819751 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:21.820468 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:21.820472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:21.820799 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:21.821233 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:21.821247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.822329 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:21.822333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:21.822760 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:21.822768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.822872 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:21.823606 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:21.823619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:21.826034 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
W20260320 02:59:21.826574 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:21.819274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:56223 (local address 127.23.132.126:39437)
0320 02:59:21.819536 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:21.819541 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:21.819783 (+ 242us) server_negotiation.cc:408] Connection header received
0320 02:59:21.819836 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:21.819839 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:21.820086 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:21.820350 (+ 264us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:21.821481 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.822157 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:21.822902 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:21.823046 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:21.826416 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:56223: BlockingRecv error: recv got EOF from 127.23.132.66:56223 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Mar 20 02:59:22 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:22 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:22.045305 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:22.039712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:55003)
0320 02:59:22.040068 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:22.040082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:22.040208 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:22.040765 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:22.040768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:22.040965 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:22.041174 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:22.041180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.042021 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:22.042024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:22.042413 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:22.042420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.042522 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:22.043350 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:22.043365 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:22.045175 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":92,"threads_started":1}
W20260320 02:59:22.045687 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:22.039781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:55003 (local address 127.23.132.126:39437)
0320 02:59:22.040074 (+ 293us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:22.040078 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:22.040193 (+ 115us) server_negotiation.cc:408] Connection header received
0320 02:59:22.040373 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:22.040376 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:22.040424 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:22.040658 (+ 234us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:22.041349 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.041886 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:22.042563 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.042709 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:22.045474 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:55003: BlockingRecv error: recv got EOF from 127.23.132.67:55003 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Mar 20 02:59:22 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:22 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:22.530295 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:22.520802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:49259)
0320 02:59:22.521126 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:22.521141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:22.521265 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:22.521527 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:22.521530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:22.521870 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:22.522090 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:22.522097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.522928 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:22.522932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:22.523306 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:22.523313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.523416 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:22.528260 (+ 4844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:22.528280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:22.530133 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
W20260320 02:59:22.530411 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:22.520889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:49259 (local address 127.23.132.126:39437)
0320 02:59:22.521048 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:22.521053 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:22.521240 (+ 187us) server_negotiation.cc:408] Connection header received
0320 02:59:22.521372 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:22.521376 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:22.521421 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:22.521488 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:22.522264 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.522793 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:22.523414 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.523587 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:22.530307 (+ 6720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:49259: BlockingRecv error: recv got EOF from 127.23.132.65:49259 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Mar 20 02:59:22 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:22 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:22.834985 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:22.827419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:41613)
0320 02:59:22.827826 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:22.827846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:22.827968 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:22.828318 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:22.828321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:22.828499 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:22.828836 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:22.828843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.829732 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:22.829735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:22.830284 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:22.830296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.830424 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:22.831214 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:22.831228 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:22.834832 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":106,"threads_started":1}
W20260320 02:59:22.835275 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:22.827634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:41613 (local address 127.23.132.126:39437)
0320 02:59:22.827820 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:22.827824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:22.828007 (+ 183us) server_negotiation.cc:408] Connection header received
0320 02:59:22.828062 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:22.828065 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:22.828112 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:22.828187 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:22.828992 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.829536 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:22.830513 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:22.830865 (+ 352us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:22.835168 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:41613: BlockingRecv error: recv got EOF from 127.23.132.66:41613 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Mar 20 02:59:23 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:23 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:23.053614 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:23.046326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:33293)
0320 02:59:23.046625 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:23.046642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:23.046753 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:23.047314 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:23.047319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:23.047571 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:23.047833 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:23.047841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.048729 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:23.048733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:23.049227 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:23.049237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.049414 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:23.050579 (+ 1165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:23.050598 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:23.053453 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
W20260320 02:59:23.053771 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:23.046486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:33293 (local address 127.23.132.126:39437)
0320 02:59:23.046691 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:23.046694 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:23.046711 (+ 17us) server_negotiation.cc:408] Connection header received
0320 02:59:23.046985 (+ 274us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:23.046989 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:23.047038 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:23.047119 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:23.048036 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.048548 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:23.049437 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.049854 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:23.053605 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:33293: BlockingRecv error: recv got EOF from 127.23.132.67:33293 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Mar 20 02:59:23 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:23 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:23.539426 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:23.531207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:46237)
0320 02:59:23.531498 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:23.531512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:23.531599 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:23.532099 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:23.532102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:23.532295 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:23.532493 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:23.532499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.533693 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:23.533698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:23.534746 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:23.534760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.534893 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:23.535417 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:23.535430 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:23.539298 (+ 3868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
W20260320 02:59:23.539867 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:23.531398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:46237 (local address 127.23.132.126:39437)
0320 02:59:23.531589 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:23.531592 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:23.531607 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:59:23.531765 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:23.531772 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:23.531833 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:23.532177 (+ 344us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:23.532719 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.533507 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:23.534962 (+ 1455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.535108 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:23.539718 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:46237: BlockingRecv error: recv got EOF from 127.23.132.65:46237 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Mar 20 02:59:23 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:23 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:23.843067 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:23.836076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:43123)
0320 02:59:23.836333 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:23.836350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:23.836452 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:23.836969 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:23.836972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:23.837155 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:23.837350 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:23.837356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.838381 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:23.838384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:23.839071 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:23.839093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.839287 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:23.839994 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:23.840009 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:23.842793 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260320 02:59:23.843230 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:23.836309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:43123 (local address 127.23.132.126:39437)
0320 02:59:23.836445 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:23.836450 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:23.836465 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:59:23.836530 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:23.836534 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:23.836591 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:23.836919 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:23.837532 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.838250 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:23.839275 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:23.839477 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:23.842989 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:43123: BlockingRecv error: recv got EOF from 127.23.132.66:43123 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Mar 20 02:59:24 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:24 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.67@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:24.063025 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:24.054841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.67:40833)
0320 02:59:24.055248 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:24.055266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:24.055443 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:24.056203 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:24.056206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:24.056427 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:24.056688 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:24.056696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.057625 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:24.057630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:24.058361 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:24.058372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.058600 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:24.059314 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:24.059331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:24.062860 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":100,"threads_started":1}
W20260320 02:59:24.063161 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:24.055072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:40833 (local address 127.23.132.126:39437)
0320 02:59:24.055197 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:24.055201 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:24.055423 (+ 222us) server_negotiation.cc:408] Connection header received
0320 02:59:24.055939 (+ 516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:24.055946 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:24.056021 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:24.056112 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:24.056956 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.057466 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:24.058564 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.058730 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:24.063061 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.67:40833: BlockingRecv error: recv got EOF from 127.23.132.67:40833 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Mar 20 02:59:24 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:24 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.65@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
I20260320 02:59:24.548250 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:24.541405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.65:49881)
0320 02:59:24.541835 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:24.541850 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:24.541970 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:24.542328 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:24.542332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:24.542587 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:24.542836 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:24.542843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.543791 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:24.543801 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:24.544205 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:24.544214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.544342 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:24.545336 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:24.545350 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:24.548082 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":106,"threads_started":1}
W20260320 02:59:24.548329 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:24.541639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:49881 (local address 127.23.132.126:39437)
0320 02:59:24.541783 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:24.541815 (+ 32us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:24.541964 (+ 149us) server_negotiation.cc:408] Connection header received
0320 02:59:24.542164 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:24.542169 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:24.542220 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:24.542286 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:24.543072 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.543556 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:24.544338 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.544519 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:24.548227 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.65:49881: BlockingRecv error: recv got EOF from 127.23.132.65:49881 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Mar 20 02:59:24 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
Mar 20 02:59:24 dist-test-slave-v3hj krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.132.66@KRBTEST.COM for oryx/127.23.132.126@KRBTEST.COM, Server not found in Kerberos database
W20260320 02:59:24.851099 1516 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:24.844464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:45237 (local address 127.23.132.126:39437)
0320 02:59:24.844605 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:24.844609 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:24.844625 (+ 16us) server_negotiation.cc:408] Connection header received
0320 02:59:24.844797 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:24.844801 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:24.844855 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:24.844946 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:24.845723 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.846386 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:24.847364 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.847590 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:24.850998 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.132.66:45237: BlockingRecv error: recv got EOF from 127.23.132.66:45237 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260320 02:59:24.851104 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:24.844262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39437 (local address 127.23.132.66:45237)
0320 02:59:24.844561 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:24.844574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:24.844661 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:24.845096 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:24.845098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:24.845277 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:24.845473 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:24.845479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.846561 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:24.846565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:24.847081 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:24.847091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:24.847343 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:24.848054 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:24.848068 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:24.850832 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.132.126:39437: Server oryx/127.23.132.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260320 02:59:24.923849 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1092
I20260320 02:59:24.931201 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1246
I20260320 02:59:24.941440 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1381
I20260320 02:59:24.948266 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1020
2026-03-20T02:59:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19512 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-03-20T02:59:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:24Z Disabled control of system clock
I20260320 02:59:24.990906 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39811
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:39811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260320 02:59:25.143536 1567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:25.143822 1567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:25.143872 1567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:25.147668 1567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:25.147766 1567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:25.147784 1567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:25.147802 1567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:25.147818 1567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:25.153321 1567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:39811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:39811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:25.155192 1567 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:25.157114 1567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:25.167443 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:25.167606 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:25.167799 1567 server_base.cc:1061] running on GCE node
W20260320 02:59:25.168296 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:25.168860 1567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:25.170009 1567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:25.171216 1567 hybrid_clock.cc:648] HybridClock initialized: now 1773975565171173 us; error 58 us; skew 500 ppm
I20260320 02:59:25.174431 1567 webserver.cc:492] Webserver started at http://127.23.132.126:44033/ using document root <none> and password file <none>
I20260320 02:59:25.175268 1567 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:25.175565 1567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:25.176339 1567 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:25.178628 1567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "ab9365c514d04a7396f43ff419b011ab"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "3333f5e308c19ccab33587d2b14c7bcc"
server_key_iv: "5dc48740b9eaf2c22b62fba570de3893"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.179289 1567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "ab9365c514d04a7396f43ff419b011ab"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "3333f5e308c19ccab33587d2b14c7bcc"
server_key_iv: "5dc48740b9eaf2c22b62fba570de3893"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.184194 1567 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260320 02:59:25.187386 1582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.189277 1567 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 02:59:25.189427 1567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "ab9365c514d04a7396f43ff419b011ab"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "3333f5e308c19ccab33587d2b14c7bcc"
server_key_iv: "5dc48740b9eaf2c22b62fba570de3893"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.189536 1567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:25.207702 1567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:25.208464 1567 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:25.208647 1567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:25.223098 1635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:39811 every 8 connection(s)
I20260320 02:59:25.223282 1567 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:39811
I20260320 02:59:25.224435 1567 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:59:25.227741 1636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:25.230417 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1567
I20260320 02:59:25.230525 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:59:25.230839 24081 external_mini_cluster.cc:1468] Setting key 1919dfc922ebb6e0991fadf89b6651e6
I20260320 02:59:25.235308 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab: Bootstrap starting.
I20260320 02:59:25.237774 1636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:25.238533 1636 log.cc:826] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:25.240454 1636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab: No bootstrap required, opened a new log
I20260320 02:59:25.244319 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:25.232639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.23.132.126:39811)
0320 02:59:25.233607 (+ 968us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:25.233636 (+ 29us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:25.233757 (+ 121us) server_negotiation.cc:408] Connection header received
0320 02:59:25.235617 (+ 1860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:25.235647 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:25.236014 (+ 367us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:25.236288 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:25.237401 (+ 1113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.240070 (+ 2669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:25.241011 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.241236 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:25.241944 (+ 708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:25.241966 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:25.241977 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:25.242336 (+ 359us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:25.242380 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:25.242393 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:25.242481 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:25.243217 (+ 736us) server_negotiation.cc:300] Negotiation successful
0320 02:59:25.243658 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
I20260320 02:59:25.244203 1636 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9365c514d04a7396f43ff419b011ab" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39811 } }
I20260320 02:59:25.244449 1636 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:25.244509 1636 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9365c514d04a7396f43ff419b011ab, State: Initialized, Role: FOLLOWER
I20260320 02:59:25.244995 1636 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9365c514d04a7396f43ff419b011ab" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39811 } }
I20260320 02:59:25.245160 1636 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:59:25.245221 1636 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:59:25.245337 1636 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:25.246686 1636 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9365c514d04a7396f43ff419b011ab" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39811 } }
I20260320 02:59:25.247081 1636 leader_election.cc:304] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9365c514d04a7396f43ff419b011ab; no voters:
I20260320 02:59:25.247375 1636 leader_election.cc:290] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:59:25.247728 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9365c514d04a7396f43ff419b011ab" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39811 } } }
I20260320 02:59:25.247794 1641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:25.247843 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [sys.catalog]: This master's current role is: FOLLOWER
I20260320 02:59:25.248171 1641 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [term 1 LEADER]: Becoming Leader. State: Replica: ab9365c514d04a7396f43ff419b011ab, State: Running, Role: LEADER
I20260320 02:59:25.249121 1641 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9365c514d04a7396f43ff419b011ab" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39811 } }
I20260320 02:59:25.248528 1636 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:59:25.252544 1643 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9365c514d04a7396f43ff419b011ab. Latest consensus state: current_term: 1 leader_uuid: "ab9365c514d04a7396f43ff419b011ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9365c514d04a7396f43ff419b011ab" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 39811 } } }
I20260320 02:59:25.253047 1643 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:25.254289 1650 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:59:25.259317 1650 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:59:25.269505 1650 catalog_manager.cc:1357] Generated new cluster ID: 65fb27bd9e8946f3b7ffb9533f60d61f
I20260320 02:59:25.269614 1650 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:59:25.301525 1650 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:59:25.303817 1650 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:59:25.316812 1650 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ab9365c514d04a7396f43ff419b011ab: Generated new TSK 0
I20260320 02:59:25.317669 1650 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:59:25.322893 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39811
--builtin_ntp_servers=127.23.132.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260320 02:59:25.526121 1660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:25.526486 1660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:25.526559 1660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:25.531262 1660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:25.532562 1660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:25.532830 1660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:59:25.541527 1660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:25.543016 1660 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:25.544194 1660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:25.553632 1666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:25.554140 1665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:25.554525 1660 server_base.cc:1061] running on GCE node
W20260320 02:59:25.554766 1668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:25.556211 1660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:25.557420 1660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:25.560246 1660 hybrid_clock.cc:648] HybridClock initialized: now 1773975565560234 us; error 30 us; skew 500 ppm
I20260320 02:59:25.562559 1660 webserver.cc:492] Webserver started at http://127.23.132.65:44813/ using document root <none> and password file <none>
I20260320 02:59:25.563238 1660 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:25.563302 1660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:25.563491 1660 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:25.565429 1660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "8711d56337b84b5ebb70ab0e6bfdd4f2"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "b683f67c32b93fc59ddf7690daada060"
server_key_iv: "dcdf6798a6f78d2226e3a5bd088b621d"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.565953 1660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "8711d56337b84b5ebb70ab0e6bfdd4f2"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "b683f67c32b93fc59ddf7690daada060"
server_key_iv: "dcdf6798a6f78d2226e3a5bd088b621d"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.569877 1660 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:25.572793 1674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.574668 1660 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260320 02:59:25.574795 1660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "8711d56337b84b5ebb70ab0e6bfdd4f2"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "b683f67c32b93fc59ddf7690daada060"
server_key_iv: "dcdf6798a6f78d2226e3a5bd088b621d"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.575480 1660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:25.589932 1660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:25.590641 1660 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:25.590818 1660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:25.591625 1660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:25.594192 1660 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:25.594257 1660 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.594316 1660 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:25.594347 1660 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.613238 1788 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:35949 every 8 connection(s)
I20260320 02:59:25.613274 1660 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:35949
I20260320 02:59:25.615010 1660 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:59:25.616191 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1660
I20260320 02:59:25.616328 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:59:25.616591 24081 external_mini_cluster.cc:1468] Setting key 9ca9dc56189315efb7f55cbaf0878a4a
I20260320 02:59:25.619828 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39811
--builtin_ntp_servers=127.23.132.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260320 02:59:25.626560 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:25.617906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:53493 (local address 127.23.132.126:39811)
0320 02:59:25.618136 (+ 230us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:25.618140 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:25.618982 (+ 842us) server_negotiation.cc:408] Connection header received
0320 02:59:25.619853 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:25.619856 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:25.619911 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:25.620454 (+ 543us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:25.621648 (+ 1194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.622412 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:25.623613 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.623839 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:25.625606 (+ 1767us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:25.625634 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:25.625638 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:25.625658 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:25.625685 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:25.625696 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:25.626088 (+ 392us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:25.626297 (+ 209us) server_negotiation.cc:300] Negotiation successful
0320 02:59:25.626386 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260320 02:59:25.627346 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:25.618052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39811 (local address 127.23.132.65:53493)
0320 02:59:25.618839 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:25.618878 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:25.619671 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:25.620522 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:25.620533 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:25.620630 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:25.621426 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:25.621445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.622659 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:25.622664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:25.623166 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:25.623178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.623831 (+ 653us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:25.625307 (+ 1476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:59:25.625309 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:25.625347 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:59:25.625416 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:25.626118 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:25.626126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:25.626131 (+ 5us) client_negotiation.cc:769] Sending connection context
0320 02:59:25.626278 (+ 147us) client_negotiation.cc:240] Negotiation successful
0320 02:59:25.626533 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260320 02:59:25.628679 1789 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39811
I20260320 02:59:25.629031 1789 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:25.629829 1789 heartbeater.cc:507] Master 127.23.132.126:39811 requested a full tablet report, sending...
I20260320 02:59:25.633080 1599 ts_manager.cc:194] Registered new tserver with Master: 8711d56337b84b5ebb70ab0e6bfdd4f2 (127.23.132.65:35949)
I20260320 02:59:25.637028 1599 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.23.132.65:53493
W20260320 02:59:25.763441 1792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:25.763701 1792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:25.763753 1792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:25.768941 1792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:25.769028 1792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:25.769109 1792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:59:25.776674 1792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:25.779058 1792 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:25.780794 1792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:25.789609 1800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:25.789690 1797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:25.790352 1798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:25.790455 1792 server_base.cc:1061] running on GCE node
I20260320 02:59:25.791366 1792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:25.792126 1792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:25.793319 1792 hybrid_clock.cc:648] HybridClock initialized: now 1773975565793245 us; error 95 us; skew 500 ppm
I20260320 02:59:25.796059 1792 webserver.cc:492] Webserver started at http://127.23.132.66:43373/ using document root <none> and password file <none>
I20260320 02:59:25.796869 1792 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:25.797127 1792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:25.797708 1792 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:25.800261 1792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "14d5d42e8ba1480b92f95de9bb39d6a4"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "fa4ecb80e655ef024a23b43e4189b253"
server_key_iv: "7d9df320bf3bbf5a5937568185c171d7"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.800977 1792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "14d5d42e8ba1480b92f95de9bb39d6a4"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "fa4ecb80e655ef024a23b43e4189b253"
server_key_iv: "7d9df320bf3bbf5a5937568185c171d7"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.805163 1792 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260320 02:59:25.807854 1806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.809149 1792 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:25.809267 1792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "14d5d42e8ba1480b92f95de9bb39d6a4"
format_stamp: "Formatted at 2026-03-20 02:59:25 on dist-test-slave-v3hj"
server_key: "fa4ecb80e655ef024a23b43e4189b253"
server_key_iv: "7d9df320bf3bbf5a5937568185c171d7"
server_key_version: "encryptionkey@0"
I20260320 02:59:25.809388 1792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:25.840658 1792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:25.842288 1792 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:25.842510 1792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:25.843294 1792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:25.844847 1792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:25.844915 1792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.844959 1792 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:25.844972 1792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:25.862736 1792 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:35555
I20260320 02:59:25.862767 1921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:35555 every 8 connection(s)
I20260320 02:59:25.864248 1792 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:59:25.870325 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1792
I20260320 02:59:25.870512 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:59:25.870929 24081 external_mini_cluster.cc:1468] Setting key d064e1aacc7fc52860099e146ba39879
I20260320 02:59:25.873268 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39811
--builtin_ntp_servers=127.23.132.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260320 02:59:25.878382 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:25.867041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:51039 (local address 127.23.132.126:39811)
0320 02:59:25.867272 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:25.867276 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:25.868293 (+ 1017us) server_negotiation.cc:408] Connection header received
0320 02:59:25.869165 (+ 872us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:25.869169 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:25.869231 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:25.869307 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:25.870431 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.871625 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:25.872586 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.872841 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:25.875565 (+ 2724us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:25.875600 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:25.875603 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:25.875632 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:25.875667 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:25.875678 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:25.876229 (+ 551us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:25.877421 (+ 1192us) server_negotiation.cc:300] Negotiation successful
0320 02:59:25.877528 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260320 02:59:25.879407 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:25.867349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39811 (local address 127.23.132.66:51039)
0320 02:59:25.868128 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:25.868165 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:25.868969 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:25.869480 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:25.869489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:25.869565 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:25.870219 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:25.870233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.871830 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:25.871834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:25.872380 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:25.872391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:25.872590 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:25.875102 (+ 2512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:59:25.875104 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:25.875150 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:59:25.875221 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:25.876237 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:25.876247 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:25.876254 (+ 7us) client_negotiation.cc:769] Sending connection context
0320 02:59:25.877418 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0320 02:59:25.878229 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":129,"threads_started":1}
I20260320 02:59:25.882628 1922 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39811
I20260320 02:59:25.883074 1922 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:25.884152 1922 heartbeater.cc:507] Master 127.23.132.126:39811 requested a full tablet report, sending...
I20260320 02:59:25.885690 1599 ts_manager.cc:194] Registered new tserver with Master: 14d5d42e8ba1480b92f95de9bb39d6a4 (127.23.132.66:35555)
I20260320 02:59:25.886554 1599 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.23.132.66:51039
W20260320 02:59:26.006480 1925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:26.006750 1925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:26.006806 1925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:26.010798 1925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:26.010885 1925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:26.010962 1925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:59:26.017249 1925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:39811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:26.018662 1925 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:26.020416 1925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:26.029810 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:26.029810 1933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:26.030508 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:26.031647 1925 server_base.cc:1061] running on GCE node
I20260320 02:59:26.032469 1925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:26.033139 1925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:26.034478 1925 hybrid_clock.cc:648] HybridClock initialized: now 1773975566034350 us; error 142 us; skew 500 ppm
I20260320 02:59:26.037181 1925 webserver.cc:492] Webserver started at http://127.23.132.67:35305/ using document root <none> and password file <none>
I20260320 02:59:26.038017 1925 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:26.038098 1925 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:26.038329 1925 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:26.040159 1925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "153499baa3d7422fb086ddf175b4b63d"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "4caf0cbc10e7a1d81a177bcdeb87dacd"
server_key_iv: "5585ba49771b21ea74318755fe5591cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.040642 1925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "153499baa3d7422fb086ddf175b4b63d"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "4caf0cbc10e7a1d81a177bcdeb87dacd"
server_key_iv: "5585ba49771b21ea74318755fe5591cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.045079 1925 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260320 02:59:26.049989 1939 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.051741 1925 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:26.051920 1925 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "153499baa3d7422fb086ddf175b4b63d"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "4caf0cbc10e7a1d81a177bcdeb87dacd"
server_key_iv: "5585ba49771b21ea74318755fe5591cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.052172 1925 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:26.075891 1925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:26.076654 1925 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:26.077098 1925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:26.078120 1925 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:26.079566 1925 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:26.079619 1925 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.080029 1925 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:26.080058 1925 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.098672 2055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:40107 every 8 connection(s)
I20260320 02:59:26.098644 1925 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:40107
I20260320 02:59:26.100291 1925 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:59:26.103725 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 1925
I20260320 02:59:26.103852 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:59:26.104107 24081 external_mini_cluster.cc:1468] Setting key 668526963acd8bf2303d51e7c1adf0e7
I20260320 02:59:26.112238 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.103025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:36435 (local address 127.23.132.126:39811)
0320 02:59:26.103226 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:26.103231 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:26.104620 (+ 1389us) server_negotiation.cc:408] Connection header received
0320 02:59:26.105618 (+ 998us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:26.105622 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:26.105674 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:26.105756 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:26.107480 (+ 1724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:26.108035 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:26.109040 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:26.109198 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:26.111316 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:26.111359 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:26.111362 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:26.111384 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:26.111415 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:26.111425 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:26.111524 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:26.112012 (+ 488us) server_negotiation.cc:300] Negotiation successful
0320 02:59:26.112094 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260320 02:59:26.113037 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.103237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:39811 (local address 127.23.132.67:36435)
0320 02:59:26.104263 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:26.104299 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:26.105435 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:26.106103 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:26.106115 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:26.106221 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:26.107236 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:26.107267 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:26.108202 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:26.108206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:26.108674 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:26.108684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:26.109141 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:26.110935 (+ 1794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:59:26.110936 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:26.110984 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:59:26.111071 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:26.111613 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:26.111624 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:26.111631 (+ 7us) client_negotiation.cc:769] Sending connection context
0320 02:59:26.111867 (+ 236us) client_negotiation.cc:240] Negotiation successful
0320 02:59:26.112149 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":135,"threads_started":1}
I20260320 02:59:26.114542 2056 heartbeater.cc:344] Connected to a master server at 127.23.132.126:39811
I20260320 02:59:26.114826 2056 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:26.115854 2056 heartbeater.cc:507] Master 127.23.132.126:39811 requested a full tablet report, sending...
I20260320 02:59:26.118031 1599 ts_manager.cc:194] Registered new tserver with Master: 153499baa3d7422fb086ddf175b4b63d (127.23.132.67:40107)
I20260320 02:59:26.119088 1599 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.23.132.67:36435
I20260320 02:59:26.120328 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260320 02:59:26.124392 1639 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:26.122819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.23.132.126:39811)
0320 02:59:26.123012 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:26.123016 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:26.123125 (+ 109us) server_negotiation.cc:408] Connection header received
0320 02:59:26.123390 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:26.123401 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:26.123460 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:26.123777 (+ 317us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:26.124083 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33718: BlockingRecv error: recv got EOF from 127.0.0.1:33718 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260320 02:59:26.125134 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1660
I20260320 02:59:26.133572 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1792
I20260320 02:59:26.140470 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1925
I20260320 02:59:26.147809 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 1567
2026-03-20T02:59:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1187 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-03-20T02:59:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:26Z Disabled control of system clock
I20260320 02:59:26.179917 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:46669
--webserver_interface=127.23.132.126
--webserver_port=0
--builtin_ntp_servers=127.23.132.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:46669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260320 02:59:26.309100 2068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:26.309360 2068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:26.309414 2068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:26.314478 2068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:26.314574 2068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:26.314589 2068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:26.314607 2068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:26.314623 2068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:26.320017 2068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:46669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:46669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:26.321179 2068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:26.322403 2068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:26.328529 2073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:26.328529 2076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:26.328529 2074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:26.329861 2068 server_base.cc:1061] running on GCE node
I20260320 02:59:26.330610 2068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:26.331895 2068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:26.333148 2068 hybrid_clock.cc:648] HybridClock initialized: now 1773975566333056 us; error 106 us; skew 500 ppm
I20260320 02:59:26.336151 2068 webserver.cc:492] Webserver started at http://127.23.132.126:38951/ using document root <none> and password file <none>
I20260320 02:59:26.336750 2068 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:26.336817 2068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:26.337006 2068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:26.339453 2068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "ee4424a876a5439c8895fb1617c5ae6a"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "adb128e33f2b2a2c69e8b6e339fae8fb"
server_key_iv: "eae1ae9e31178561012ed227b509c734"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.340286 2068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "ee4424a876a5439c8895fb1617c5ae6a"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "adb128e33f2b2a2c69e8b6e339fae8fb"
server_key_iv: "eae1ae9e31178561012ed227b509c734"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.347402 2068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260320 02:59:26.350657 2083 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.352643 2068 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:26.353129 2068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "ee4424a876a5439c8895fb1617c5ae6a"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "adb128e33f2b2a2c69e8b6e339fae8fb"
server_key_iv: "eae1ae9e31178561012ed227b509c734"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.353300 2068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:26.362309 2068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:26.363016 2068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:26.363183 2068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:26.373672 2138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:46669 every 8 connection(s)
I20260320 02:59:26.373677 2068 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:46669
I20260320 02:59:26.375020 2068 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:59:26.377547 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2068
I20260320 02:59:26.377684 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:59:26.378103 24081 external_mini_cluster.cc:1468] Setting key 879b02c91501000643c29cc913d0c2d1
I20260320 02:59:26.380254 2139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:26.385959 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.380053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.23.132.126:46669)
0320 02:59:26.381658 (+ 1605us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:26.381669 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:26.381711 (+ 42us) server_negotiation.cc:408] Connection header received
0320 02:59:26.382531 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:26.382555 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:26.382960 (+ 405us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:26.383457 (+ 497us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:26.384493 (+ 1036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:26.384500 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:26.384513 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:26.384526 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:26.384576 (+ 50us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:26.384590 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:26.384774 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:26.384943 (+ 169us) server_negotiation.cc:300] Negotiation successful
0320 02:59:26.385185 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":478,"threads_started":1}
I20260320 02:59:26.387730 2139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a: Bootstrap starting.
I20260320 02:59:26.389945 2139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:26.390681 2139 log.cc:826] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:26.393206 2139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a: No bootstrap required, opened a new log
I20260320 02:59:26.396880 2139 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4424a876a5439c8895fb1617c5ae6a" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46669 } }
I20260320 02:59:26.397111 2139 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:26.397150 2139 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4424a876a5439c8895fb1617c5ae6a, State: Initialized, Role: FOLLOWER
I20260320 02:59:26.397549 2139 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4424a876a5439c8895fb1617c5ae6a" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46669 } }
I20260320 02:59:26.397676 2139 raft_consensus.cc:399] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:59:26.397715 2139 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:59:26.397845 2139 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:26.400012 2139 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4424a876a5439c8895fb1617c5ae6a" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46669 } }
I20260320 02:59:26.400413 2139 leader_election.cc:304] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee4424a876a5439c8895fb1617c5ae6a; no voters:
I20260320 02:59:26.400743 2139 leader_election.cc:290] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:59:26.401104 2144 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:26.401546 2144 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [term 1 LEADER]: Becoming Leader. State: Replica: ee4424a876a5439c8895fb1617c5ae6a, State: Running, Role: LEADER
I20260320 02:59:26.401952 2144 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4424a876a5439c8895fb1617c5ae6a" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46669 } }
I20260320 02:59:26.402922 2139 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:59:26.404965 2146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4424a876a5439c8895fb1617c5ae6a. Latest consensus state: current_term: 1 leader_uuid: "ee4424a876a5439c8895fb1617c5ae6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4424a876a5439c8895fb1617c5ae6a" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46669 } } }
I20260320 02:59:26.405292 2146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:26.404965 2145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee4424a876a5439c8895fb1617c5ae6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee4424a876a5439c8895fb1617c5ae6a" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 46669 } } }
I20260320 02:59:26.405592 2145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a [sys.catalog]: This master's current role is: LEADER
W20260320 02:59:26.409621 2158 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:59:26.409722 2158 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:59:26.409837 2159 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:59:26.410851 2159 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:59:26.419368 2159 catalog_manager.cc:1357] Generated new cluster ID: e7ab11419acb4e4cbc54a45555d4b0ef
I20260320 02:59:26.419507 2159 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:59:26.476819 2159 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:59:26.477205 2159 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:59:26.488046 2159 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ee4424a876a5439c8895fb1617c5ae6a: Generated new TSK 0
I20260320 02:59:26.488808 2159 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260320 02:59:26.505302 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46669
--builtin_ntp_servers=127.23.132.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260320 02:59:26.631314 2163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:26.631868 2163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:26.632200 2163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:26.637388 2163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:26.637465 2163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:26.637544 2163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:59:26.642217 2163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:26.643553 2163 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:26.644701 2163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:26.653228 2171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:26.653753 2163 server_base.cc:1061] running on GCE node
W20260320 02:59:26.653771 2169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:26.653873 2168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:26.655014 2163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:26.656078 2163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:26.659715 2163 hybrid_clock.cc:648] HybridClock initialized: now 1773975566659689 us; error 37 us; skew 500 ppm
I20260320 02:59:26.662362 2163 webserver.cc:492] Webserver started at http://127.23.132.65:37787/ using document root <none> and password file <none>
I20260320 02:59:26.662973 2163 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:26.663031 2163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:26.663229 2163 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:26.665673 2163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "f48932a11dbe4b4bbf46de928f9e9316"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "4432d8879c420513a9e0f53c01ab64bc"
server_key_iv: "847be7b025f4785d2601931d6e06b432"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.666620 2163 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "f48932a11dbe4b4bbf46de928f9e9316"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "4432d8879c420513a9e0f53c01ab64bc"
server_key_iv: "847be7b025f4785d2601931d6e06b432"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.671672 2163 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260320 02:59:26.674448 2177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.676246 2163 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260320 02:59:26.676851 2163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "f48932a11dbe4b4bbf46de928f9e9316"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "4432d8879c420513a9e0f53c01ab64bc"
server_key_iv: "847be7b025f4785d2601931d6e06b432"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.677124 2163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:26.683895 2163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:26.684916 2163 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:26.685182 2163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:26.686224 2163 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:26.687604 2163 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:26.687680 2163 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.687739 2163 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:26.687772 2163 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.708360 2290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:41957 every 8 connection(s)
I20260320 02:59:26.708391 2163 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:41957
I20260320 02:59:26.709540 2163 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:59:26.714486 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2163
I20260320 02:59:26.714623 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:59:26.714903 24081 external_mini_cluster.cc:1468] Setting key 6e18f2adb6682f3983cadf162b814e96
I20260320 02:59:26.717131 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.711887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:48379 (local address 127.23.132.126:46669)
0320 02:59:26.712055 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:26.712058 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:26.713125 (+ 1067us) server_negotiation.cc:408] Connection header received
0320 02:59:26.714751 (+ 1626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:26.714756 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:26.714828 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:26.714922 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:26.716646 (+ 1724us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:26.716651 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:26.716654 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:26.716669 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:26.716697 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:26.716705 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:26.716789 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:26.716956 (+ 167us) server_negotiation.cc:300] Negotiation successful
0320 02:59:26.717001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260320 02:59:26.717185 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46669
--builtin_ntp_servers=127.23.132.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260320 02:59:26.717907 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.711917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46669 (local address 127.23.132.65:48379)
0320 02:59:26.712906 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:26.712957 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:26.714439 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:26.715336 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:26.715350 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:26.715589 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:26.716176 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:59:26.716178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:26.716211 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:59:26.716276 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:26.716792 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:26.716799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:26.716805 (+ 6us) client_negotiation.cc:769] Sending connection context
0320 02:59:26.716999 (+ 194us) client_negotiation.cc:240] Negotiation successful
0320 02:59:26.717235 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":126,"threads_started":1}
I20260320 02:59:26.720243 2291 heartbeater.cc:344] Connected to a master server at 127.23.132.126:46669
I20260320 02:59:26.720474 2291 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:26.720969 2291 heartbeater.cc:507] Master 127.23.132.126:46669 requested a full tablet report, sending...
I20260320 02:59:26.722666 2100 ts_manager.cc:194] Registered new tserver with Master: f48932a11dbe4b4bbf46de928f9e9316 (127.23.132.65:41957)
W20260320 02:59:26.862571 2294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:26.862991 2294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:26.863235 2294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:26.868943 2294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:26.869027 2294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:26.869115 2294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:59:26.874462 2294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:26.875963 2294 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:26.877180 2294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:26.885566 2299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:26.885571 2300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:26.886011 2294 server_base.cc:1061] running on GCE node
W20260320 02:59:26.885991 2302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:26.887009 2294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:26.887851 2294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:26.889174 2294 hybrid_clock.cc:648] HybridClock initialized: now 1773975566889143 us; error 50 us; skew 500 ppm
I20260320 02:59:26.892068 2294 webserver.cc:492] Webserver started at http://127.23.132.66:41801/ using document root <none> and password file <none>
I20260320 02:59:26.892693 2294 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:26.892766 2294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:26.892962 2294 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:26.894800 2294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "0576f82e9f324e54ac6b8a414be92838"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "62644b7bdca89d345cdfee16f7d19616"
server_key_iv: "e059d7aeb7b04f8eea9affe364c85d7a"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.895284 2294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "0576f82e9f324e54ac6b8a414be92838"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "62644b7bdca89d345cdfee16f7d19616"
server_key_iv: "e059d7aeb7b04f8eea9affe364c85d7a"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.899425 2294 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:26.902616 2308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.904913 2294 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260320 02:59:26.905074 2294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "0576f82e9f324e54ac6b8a414be92838"
format_stamp: "Formatted at 2026-03-20 02:59:26 on dist-test-slave-v3hj"
server_key: "62644b7bdca89d345cdfee16f7d19616"
server_key_iv: "e059d7aeb7b04f8eea9affe364c85d7a"
server_key_version: "encryptionkey@0"
I20260320 02:59:26.905205 2294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:26.913594 2294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:26.914451 2294 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:26.914616 2294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:26.915177 2294 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:26.916551 2294 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:26.916656 2294 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.916713 2294 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:26.916728 2294 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:26.933079 2294 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:45239
I20260320 02:59:26.933605 2421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:45239 every 8 connection(s)
I20260320 02:59:26.934968 2294 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:59:26.937767 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2294
I20260320 02:59:26.938287 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:59:26.938587 24081 external_mini_cluster.cc:1468] Setting key 484e6151f682b71e76f5c43cddfbbc3c
I20260320 02:59:26.941884 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46669
--builtin_ntp_servers=127.23.132.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260320 02:59:26.943941 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.938366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:40321 (local address 127.23.132.126:46669)
0320 02:59:26.938551 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:26.938556 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:26.940259 (+ 1703us) server_negotiation.cc:408] Connection header received
0320 02:59:26.941306 (+ 1047us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:26.941310 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:26.941358 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:26.941438 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:26.943299 (+ 1861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:26.943312 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:26.943316 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:26.943333 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:26.943357 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:26.943369 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:26.943493 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:26.943692 (+ 199us) server_negotiation.cc:300] Negotiation successful
0320 02:59:26.943794 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260320 02:59:26.944564 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:26.938692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46669 (local address 127.23.132.66:40321)
0320 02:59:26.940103 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:26.940140 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:26.941091 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:26.941606 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:26.941614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:26.941770 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:26.942491 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:59:26.942493 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:26.942525 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:59:26.942581 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:26.943508 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:26.943518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:26.943524 (+ 6us) client_negotiation.cc:769] Sending connection context
0320 02:59:26.943727 (+ 203us) client_negotiation.cc:240] Negotiation successful
0320 02:59:26.943914 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":141,"threads_started":1}
I20260320 02:59:26.945981 2422 heartbeater.cc:344] Connected to a master server at 127.23.132.126:46669
I20260320 02:59:26.946528 2422 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:26.947098 2422 heartbeater.cc:507] Master 127.23.132.126:46669 requested a full tablet report, sending...
I20260320 02:59:26.948999 2100 ts_manager.cc:194] Registered new tserver with Master: 0576f82e9f324e54ac6b8a414be92838 (127.23.132.66:45239)
W20260320 02:59:27.078699 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:27.079017 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:27.079665 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:27.083961 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:27.084031 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:27.084113 2425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:59:27.089545 2425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:46669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:27.091711 2425 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:27.093729 2425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:27.101869 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:27.101869 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:27.102754 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:27.104116 2425 server_base.cc:1061] running on GCE node
I20260320 02:59:27.104689 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:27.105374 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:27.106702 2425 hybrid_clock.cc:648] HybridClock initialized: now 1773975567106671 us; error 29 us; skew 500 ppm
I20260320 02:59:27.110379 2425 webserver.cc:492] Webserver started at http://127.23.132.67:40227/ using document root <none> and password file <none>
I20260320 02:59:27.111007 2425 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:27.111068 2425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:27.111264 2425 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:27.113240 2425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "bd8e74bcd2244260980a68c1079062e3"
format_stamp: "Formatted at 2026-03-20 02:59:27 on dist-test-slave-v3hj"
server_key: "7f00c000cd6ec9b1a10222d114cd2135"
server_key_iv: "cab85be4949547c8a690b67bf624cbee"
server_key_version: "encryptionkey@0"
I20260320 02:59:27.113754 2425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "bd8e74bcd2244260980a68c1079062e3"
format_stamp: "Formatted at 2026-03-20 02:59:27 on dist-test-slave-v3hj"
server_key: "7f00c000cd6ec9b1a10222d114cd2135"
server_key_iv: "cab85be4949547c8a690b67bf624cbee"
server_key_version: "encryptionkey@0"
I20260320 02:59:27.122273 2425 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260320 02:59:27.125630 2439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:27.129302 2425 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260320 02:59:27.129699 2425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "bd8e74bcd2244260980a68c1079062e3"
format_stamp: "Formatted at 2026-03-20 02:59:27 on dist-test-slave-v3hj"
server_key: "7f00c000cd6ec9b1a10222d114cd2135"
server_key_iv: "cab85be4949547c8a690b67bf624cbee"
server_key_version: "encryptionkey@0"
I20260320 02:59:27.129936 2425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:27.137319 2425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:27.138726 2425 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:27.138918 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:27.139657 2425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:27.140746 2425 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:27.140797 2425 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:27.140851 2425 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:27.140878 2425 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:27.161398 2425 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:37161
I20260320 02:59:27.161437 2552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:37161 every 8 connection(s)
I20260320 02:59:27.163549 2425 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:59:27.172423 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:27.166979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:35193 (local address 127.23.132.126:46669)
0320 02:59:27.167221 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:27.167225 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:27.168608 (+ 1383us) server_negotiation.cc:408] Connection header received
0320 02:59:27.169602 (+ 994us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:27.169607 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:27.169676 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:27.170539 (+ 863us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:27.171792 (+ 1253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:27.171797 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:27.171800 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0320 02:59:27.171817 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:27.171846 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0320 02:59:27.171855 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:27.171959 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:27.172229 (+ 270us) server_negotiation.cc:300] Negotiation successful
0320 02:59:27.172292 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260320 02:59:27.173108 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2425
I20260320 02:59:27.173236 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:59:27.173313 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:27.167425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:46669 (local address 127.23.132.67:35193)
0320 02:59:27.168188 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:27.168397 (+ 209us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:27.169137 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:27.170707 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:27.170717 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:27.170808 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:27.171456 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0320 02:59:27.171458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:27.171490 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0320 02:59:27.171551 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:27.171990 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:27.172002 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:27.172006 (+ 4us) client_negotiation.cc:769] Sending connection context
0320 02:59:27.172245 (+ 239us) client_negotiation.cc:240] Negotiation successful
0320 02:59:27.172447 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":181,"threads_started":1}
I20260320 02:59:27.173511 24081 external_mini_cluster.cc:1468] Setting key 552aea2ae744e39b8b2808fb3ee70b1f
I20260320 02:59:27.175860 2553 heartbeater.cc:344] Connected to a master server at 127.23.132.126:46669
I20260320 02:59:27.176363 2553 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:27.177289 2553 heartbeater.cc:507] Master 127.23.132.126:46669 requested a full tablet report, sending...
I20260320 02:59:27.178797 2100 ts_manager.cc:194] Registered new tserver with Master: bd8e74bcd2244260980a68c1079062e3 (127.23.132.67:37161)
I20260320 02:59:27.190335 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260320 02:59:27.194586 2142 negotiation.cc:337] Failed RPC negotiation. Trace:
0320 02:59:27.192837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.23.132.126:46669)
0320 02:59:27.193142 (+ 305us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:27.193147 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:27.193264 (+ 117us) server_negotiation.cc:408] Connection header received
0320 02:59:27.193392 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:27.193396 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:27.193436 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:27.193504 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:27.193716 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47098: BlockingRecv error: recv got EOF from 127.0.0.1:47098 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
I20260320 02:59:27.196327 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2163
I20260320 02:59:27.205070 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2294
I20260320 02:59:27.212114 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2425
I20260320 02:59:27.221452 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2068
2026-03-20T02:59:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1074 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:59:27 dist-test-slave-v3hj krb5kdc[2563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:59:27 dist-test-slave-v3hj krb5kdc[2563](info): set up 2 sockets
Mar 20 02:59:27 dist-test-slave-v3hj krb5kdc[2563](info): commencing operation
krb5kdc: starting...
W20260320 02:59:29.402091 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.158s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:59:29 dist-test-slave-v3hj krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:59:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:29.615399 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:45613
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:29.758878 2579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:29.759137 2579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:29.759196 2579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:29.763943 2579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 02:59:29.764038 2579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:29.764061 2579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:29.764078 2579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 02:59:29.764096 2579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 02:59:29.769129 2579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:45613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:29.770361 2579 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:29.771210 2579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:29.777776 2587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:29.777776 2584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:29.778313 2579 server_base.cc:1061] running on GCE node
W20260320 02:59:29.778452 2585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:29.779109 2579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:29.780369 2579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:29.781596 2579 hybrid_clock.cc:648] HybridClock initialized: now 1773975569781587 us; error 60 us; skew 500 ppm
Mar 20 02:59:29 dist-test-slave-v3hj krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975569, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:29.785825 2579 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 02:59:29.787156 2579 webserver.cc:492] Webserver started at http://127.23.132.126:45251/ using document root <none> and password file <none>
I20260320 02:59:29.787730 2579 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:29.787781 2579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:29.788192 2579 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:29.790447 2579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "dd6721fda1564bbd85210f917554ad8d"
format_stamp: "Formatted at 2026-03-20 02:59:29 on dist-test-slave-v3hj"
server_key: "795a4006cf60d1ed4aa15ab1b2390c5e"
server_key_iv: "46df7cca4b84a86e220a27c876127441"
server_key_version: "encryptionkey@0"
I20260320 02:59:29.791019 2579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "dd6721fda1564bbd85210f917554ad8d"
format_stamp: "Formatted at 2026-03-20 02:59:29 on dist-test-slave-v3hj"
server_key: "795a4006cf60d1ed4aa15ab1b2390c5e"
server_key_iv: "46df7cca4b84a86e220a27c876127441"
server_key_version: "encryptionkey@0"
I20260320 02:59:29.796372 2579 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260320 02:59:29.799783 2594 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:29.801071 2579 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 02:59:29.801211 2579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "dd6721fda1564bbd85210f917554ad8d"
format_stamp: "Formatted at 2026-03-20 02:59:29 on dist-test-slave-v3hj"
server_key: "795a4006cf60d1ed4aa15ab1b2390c5e"
server_key_iv: "46df7cca4b84a86e220a27c876127441"
server_key_version: "encryptionkey@0"
I20260320 02:59:29.801342 2579 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:29.819468 2579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:29.828114 2579 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:29.828456 2579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:29.841338 2579 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:45613
I20260320 02:59:29.841511 2646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:45613 every 8 connection(s)
I20260320 02:59:29.843442 2579 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 02:59:29.844534 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2579
I20260320 02:59:29.844686 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 02:59:29.844982 24081 external_mini_cluster.cc:1468] Setting key 53706a2ce54afbc7608b709b98132674
I20260320 02:59:29.847056 2647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:29.855868 2647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: Bootstrap starting.
Mar 20 02:59:29 dist-test-slave-v3hj krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:59:29.858511 2647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:29.859237 2647 log.cc:826] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:29.861667 2647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: No bootstrap required, opened a new log
I20260320 02:59:29.864423 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:29.847419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.23.132.126:45613)
0320 02:59:29.848365 (+ 946us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:29.848374 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:29.848409 (+ 35us) server_negotiation.cc:408] Connection header received
0320 02:59:29.849119 (+ 710us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:29.849138 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:29.849422 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:29.849920 (+ 498us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:29.851090 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:29.851994 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:29.853060 (+ 1066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:29.853329 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:29.857141 (+ 3812us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:29.857186 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:29.857199 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:29.857249 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:29.860685 (+ 3436us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:29.861417 (+ 732us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:29.861427 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:29.861435 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:29.861699 (+ 264us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:29.862257 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:29.862263 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:29.862265 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:29.862820 (+ 555us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:29.863296 (+ 476us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:29.863457 (+ 161us) server_negotiation.cc:300] Negotiation successful
0320 02:59:29.863798 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":497,"threads_started":1}
I20260320 02:59:29.864827 2647 raft_consensus.cc:359] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6721fda1564bbd85210f917554ad8d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45613 } }
I20260320 02:59:29.865036 2647 raft_consensus.cc:385] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:29.865113 2647 raft_consensus.cc:740] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6721fda1564bbd85210f917554ad8d, State: Initialized, Role: FOLLOWER
I20260320 02:59:29.865567 2647 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6721fda1564bbd85210f917554ad8d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45613 } }
I20260320 02:59:29.865769 2647 raft_consensus.cc:399] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 02:59:29.865849 2647 raft_consensus.cc:493] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 02:59:29.865978 2647 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:29.867460 2647 raft_consensus.cc:515] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6721fda1564bbd85210f917554ad8d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45613 } }
I20260320 02:59:29.868399 2647 leader_election.cc:304] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6721fda1564bbd85210f917554ad8d; no voters:
I20260320 02:59:29.868729 2647 leader_election.cc:290] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 02:59:29.868856 2652 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:29.869040 2652 raft_consensus.cc:697] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [term 1 LEADER]: Becoming Leader. State: Replica: dd6721fda1564bbd85210f917554ad8d, State: Running, Role: LEADER
I20260320 02:59:29.869333 2652 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6721fda1564bbd85210f917554ad8d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45613 } }
I20260320 02:59:29.870004 2647 sys_catalog.cc:565] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [sys.catalog]: configured and running, proceeding with master startup.
I20260320 02:59:29.872095 2654 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6721fda1564bbd85210f917554ad8d. Latest consensus state: current_term: 1 leader_uuid: "dd6721fda1564bbd85210f917554ad8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6721fda1564bbd85210f917554ad8d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45613 } } }
I20260320 02:59:29.872428 2654 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [sys.catalog]: This master's current role is: LEADER
I20260320 02:59:29.872367 2653 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6721fda1564bbd85210f917554ad8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6721fda1564bbd85210f917554ad8d" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 45613 } } }
I20260320 02:59:29.872872 2653 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d [sys.catalog]: This master's current role is: LEADER
W20260320 02:59:29.880282 2668 catalog_manager.cc:1568] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260320 02:59:29.880743 2668 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260320 02:59:29.881366 2661 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 02:59:29.883075 2661 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 02:59:29.889856 2661 catalog_manager.cc:1357] Generated new cluster ID: 8e485d3b9fa34069a00cb58a55f8854a
I20260320 02:59:29.889971 2661 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 02:59:29.902463 2661 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 02:59:29.903815 2661 catalog_manager.cc:1514] Loading token signing keys...
I20260320 02:59:29.910445 2661 catalog_manager.cc:6044] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: Generated new TSK 0
I20260320 02:59:29.911245 2661 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:30.000487 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:45613
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:30.161983 2675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:30.162241 2675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:30.162318 2675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:30.166848 2675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:30.166945 2675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:30.167044 2675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 02:59:30.172435 2675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:30.174832 2675 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:30.175693 2675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:30.184689 2680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:30.184983 2683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:30.185482 2681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:30.186517 2675 server_base.cc:1061] running on GCE node
I20260320 02:59:30.187075 2675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:30.187670 2675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:30.188921 2675 hybrid_clock.cc:648] HybridClock initialized: now 1773975570188904 us; error 32 us; skew 500 ppm
Mar 20 02:59:30 dist-test-slave-v3hj krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975570, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:30.192378 2675 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 02:59:30.193536 2675 webserver.cc:492] Webserver started at http://127.23.132.65:43917/ using document root <none> and password file <none>
I20260320 02:59:30.194324 2675 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:30.194381 2675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:30.194571 2675 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:30.198582 2675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "a3afdc28ef074d02b62779a396597481"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "8ba9a0f0e7f2b89d3d85ea502c647f47"
server_key_iv: "25eedffc2c374059637131f7f3a8269a"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.199543 2675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "a3afdc28ef074d02b62779a396597481"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "8ba9a0f0e7f2b89d3d85ea502c647f47"
server_key_iv: "25eedffc2c374059637131f7f3a8269a"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.204723 2675 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260320 02:59:30.207551 2691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.208817 2675 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260320 02:59:30.208947 2675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "a3afdc28ef074d02b62779a396597481"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "8ba9a0f0e7f2b89d3d85ea502c647f47"
server_key_iv: "25eedffc2c374059637131f7f3a8269a"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.209052 2675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:30.234539 2675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:30.243480 2675 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:30.243688 2675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:30.244295 2675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:30.246392 2675 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:30.247448 2675 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260320 02:59:30.248070 2675 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:30.248134 2675 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.266944 2675 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:45353
I20260320 02:59:30.267174 2805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:45353 every 8 connection(s)
I20260320 02:59:30.268240 2675 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 02:59:30.274586 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2675
I20260320 02:59:30.274767 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 02:59:30.275879 24081 external_mini_cluster.cc:1468] Setting key a1838adacdd892b717afc07a064e556d
Mar 20 02:59:30 dist-test-slave-v3hj krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975570, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:59:30.284849 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:30.270233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:52689 (local address 127.23.132.126:45613)
0320 02:59:30.270389 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:30.270392 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:30.271198 (+ 806us) server_negotiation.cc:408] Connection header received
0320 02:59:30.272063 (+ 865us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:30.272067 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:30.272128 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:30.272201 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:30.274514 (+ 2313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.275614 (+ 1100us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:30.276562 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.276751 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:30.280502 (+ 3751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:30.280526 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:30.280528 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:30.280554 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:30.282801 (+ 2247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:30.283382 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:30.283386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:30.283387 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:30.283433 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:30.283711 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:30.283715 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:30.283716 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:30.283894 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:30.283988 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:30.284530 (+ 542us) server_negotiation.cc:300] Negotiation successful
0320 02:59:30.284679 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260320 02:59:30.286195 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:30.270446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45613 (local address 127.23.132.65:52689)
0320 02:59:30.271049 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:30.271084 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:30.271871 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:30.272358 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:30.272366 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:30.273373 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:30.274281 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:30.274299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.275774 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:30.275778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:30.276428 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:30.276440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.276728 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:30.278847 (+ 2119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:30.278882 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:30.280332 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:30.282987 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:30.282994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:30.283007 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:30.283278 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:30.283514 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:30.283517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:30.283519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:30.283613 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:30.284006 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:30.284013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:30.284282 (+ 269us) client_negotiation.cc:769] Sending connection context
0320 02:59:30.284479 (+ 197us) client_negotiation.cc:240] Negotiation successful
0320 02:59:30.284787 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":136,"threads_started":1}
I20260320 02:59:30.287915 2806 heartbeater.cc:344] Connected to a master server at 127.23.132.126:45613
I20260320 02:59:30.288200 2806 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 02:59:30.289484 2806 heartbeater.cc:507] Master 127.23.132.126:45613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
I20260320 02:59:30.294234 2611 ts_manager.cc:194] Registered new tserver with Master: a3afdc28ef074d02b62779a396597481 (127.23.132.65:45353)
I20260320 02:59:30.297966 2611 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:52689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:30.372886 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:45613
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:30.522258 2813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:30.522500 2813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:30.522562 2813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:30.526523 2813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:30.526623 2813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:30.526711 2813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 02:59:30.531649 2813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:30.535408 2813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:30.538697 2813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:30.546948 2819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:30.546948 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:30.547063 2820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:30.548116 2813 server_base.cc:1061] running on GCE node
I20260320 02:59:30.548660 2813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:30.549436 2813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:30.551040 2813 hybrid_clock.cc:648] HybridClock initialized: now 1773975570551027 us; error 27 us; skew 500 ppm
Mar 20 02:59:30 dist-test-slave-v3hj krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975570, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:30.558315 2813 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 02:59:30.560199 2813 webserver.cc:492] Webserver started at http://127.23.132.66:39433/ using document root <none> and password file <none>
I20260320 02:59:30.561431 2813 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:30.561923 2813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:30.562518 2813 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:30.564383 2813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "c11311dd8feb41cb8377ee5011f9b417"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "9566bf35029cd864100979ec4a68c537"
server_key_iv: "b171e8d72c00c21f260384b08598f5cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.564855 2813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "c11311dd8feb41cb8377ee5011f9b417"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "9566bf35029cd864100979ec4a68c537"
server_key_iv: "b171e8d72c00c21f260384b08598f5cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.569697 2813 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260320 02:59:30.573398 2829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.575019 2813 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260320 02:59:30.575804 2813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "c11311dd8feb41cb8377ee5011f9b417"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "9566bf35029cd864100979ec4a68c537"
server_key_iv: "b171e8d72c00c21f260384b08598f5cb"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.575945 2813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:30.609579 2813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:30.615882 2813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:30.617410 2813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:30.619200 2813 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:30.620908 2813 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:30.620977 2813 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.621037 2813 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:30.621086 2813 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.643967 2813 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:37347
I20260320 02:59:30.644061 2944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:37347 every 8 connection(s)
I20260320 02:59:30.644964 2813 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 02:59:30.653998 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2813
I20260320 02:59:30.654121 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 02:59:30.654760 24081 external_mini_cluster.cc:1468] Setting key bf4c951f28b6f24e3a2353c66042ef1d
Mar 20 02:59:30 dist-test-slave-v3hj krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975570, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:59:30.666594 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:30.647757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.66:54969 (local address 127.23.132.126:45613)
0320 02:59:30.648121 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:30.648124 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:30.648993 (+ 869us) server_negotiation.cc:408] Connection header received
0320 02:59:30.649845 (+ 852us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:30.649848 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:30.649901 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:30.649981 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:30.651571 (+ 1590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.652335 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:30.654355 (+ 2020us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.654953 (+ 598us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:30.662350 (+ 7397us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:30.662375 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:30.662378 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:30.662410 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:30.663961 (+ 1551us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:30.664713 (+ 752us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:30.664716 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:30.664718 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:30.664770 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:30.665164 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:30.665167 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:30.665168 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:30.665347 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:30.665447 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:30.666329 (+ 882us) server_negotiation.cc:300] Negotiation successful
0320 02:59:30.666462 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260320 02:59:30.667392 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:30.648256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45613 (local address 127.23.132.66:54969)
0320 02:59:30.648823 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:30.648857 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:30.649607 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:30.650362 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:30.650370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:30.650818 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:30.651365 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:30.651378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.652776 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:30.652783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:30.653331 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:30.653348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.655139 (+ 1791us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:30.658286 (+ 3147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:30.658315 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:30.662063 (+ 3748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:30.664121 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:30.664129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:30.664144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:30.664509 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:30.664866 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:30.664869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:30.664871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:30.665020 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:30.665470 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:30.665477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:30.665868 (+ 391us) client_negotiation.cc:769] Sending connection context
0320 02:59:30.666140 (+ 272us) client_negotiation.cc:240] Negotiation successful
0320 02:59:30.666472 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.23.132.67@KRBTEST.COM; defaulting to no policy
I20260320 02:59:30.669291 2945 heartbeater.cc:344] Connected to a master server at 127.23.132.126:45613
I20260320 02:59:30.669955 2945 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:30.670936 2945 heartbeater.cc:507] Master 127.23.132.126:45613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.67@KRBTEST.COM" created.
I20260320 02:59:30.673345 2611 ts_manager.cc:194] Registered new tserver with Master: c11311dd8feb41cb8377ee5011f9b417 (127.23.132.66:37347)
I20260320 02:59:30.674216 2611 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.66@KRBTEST.COM'} at 127.23.132.66:54969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 02:59:30.731721 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.67:0
--local_ip_for_outbound_sockets=127.23.132.67
--webserver_interface=127.23.132.67
--webserver_port=0
--tserver_master_addrs=127.23.132.126:45613
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 02:59:30.865777 2952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 02:59:30.866099 2952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 02:59:30.866163 2952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 02:59:30.871063 2952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 02:59:30.871145 2952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 02:59:30.871244 2952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.67
I20260320 02:59:30.876911 2952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.132.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.67
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 02:59:30.878319 2952 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 02:59:30.879199 2952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 02:59:30.887630 2957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:30.887641 2958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 02:59:30.887646 2960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 02:59:30.889149 2952 server_base.cc:1061] running on GCE node
I20260320 02:59:30.889572 2952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 02:59:30.890246 2952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 02:59:30.891440 2952 hybrid_clock.cc:648] HybridClock initialized: now 1773975570891371 us; error 83 us; skew 500 ppm
Mar 20 02:59:30 dist-test-slave-v3hj krb5kdc[2563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975570, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 02:59:30.895917 2952 init.cc:377] Logged in from keytab as kudu/127.23.132.67@KRBTEST.COM (short username kudu)
I20260320 02:59:30.897650 2952 webserver.cc:492] Webserver started at http://127.23.132.67:41719/ using document root <none> and password file <none>
I20260320 02:59:30.898491 2952 fs_manager.cc:362] Metadata directory not provided
I20260320 02:59:30.898603 2952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 02:59:30.899210 2952 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260320 02:59:30.901424 2952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/instance:
uuid: "901a49e207444ca29b9d932a2486f809"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "55e864e0a0ccecd475783a29b523de3c"
server_key_iv: "6850a90de209d5d61deef1dffc6a49fa"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.901958 2952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance:
uuid: "901a49e207444ca29b9d932a2486f809"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "55e864e0a0ccecd475783a29b523de3c"
server_key_iv: "6850a90de209d5d61deef1dffc6a49fa"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.906967 2952 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260320 02:59:30.909544 2967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.910923 2952 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260320 02:59:30.911038 2952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
uuid: "901a49e207444ca29b9d932a2486f809"
format_stamp: "Formatted at 2026-03-20 02:59:30 on dist-test-slave-v3hj"
server_key: "55e864e0a0ccecd475783a29b523de3c"
server_key_iv: "6850a90de209d5d61deef1dffc6a49fa"
server_key_version: "encryptionkey@0"
I20260320 02:59:30.911243 2952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 02:59:30.941843 2952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 02:59:30.944995 2952 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 02:59:30.945196 2952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 02:59:30.945832 2952 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 02:59:30.947024 2952 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 02:59:30.947088 2952 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.947144 2952 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 02:59:30.947173 2952 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 02:59:30.970485 2952 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.67:34583
I20260320 02:59:30.970746 3081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.67:34583 every 8 connection(s)
I20260320 02:59:30.972159 2952 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/data/info.pb
I20260320 02:59:30.982297 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 2952
I20260320 02:59:30.982420 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/instance
I20260320 02:59:30.982666 24081 external_mini_cluster.cc:1468] Setting key 7fc24eca8ae6c6fe5f5210039f09f416
Mar 20 02:59:30 dist-test-slave-v3hj krb5kdc[2563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975570, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.67@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 02:59:30.994958 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:30.974999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:37951 (local address 127.23.132.126:45613)
0320 02:59:30.975174 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:30.975177 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:30.976073 (+ 896us) server_negotiation.cc:408] Connection header received
0320 02:59:30.978455 (+ 2382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:30.978459 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:30.978527 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:30.978603 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:30.979997 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.980498 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:30.981406 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.981592 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:30.985984 (+ 4392us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:30.986003 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:30.986006 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:30.986037 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:30.990063 (+ 4026us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:30.991105 (+ 1042us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:30.991110 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:30.991111 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:30.991171 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:30.991944 (+ 773us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:30.991947 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:30.991949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:30.992133 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:30.992425 (+ 292us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:30.994529 (+ 2104us) server_negotiation.cc:300] Negotiation successful
0320 02:59:30.994746 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260320 02:59:30.996073 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:30.975233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:45613 (local address 127.23.132.67:37951)
0320 02:59:30.975853 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:30.975893 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:30.978205 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:30.978812 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:30.978823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:30.979268 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0320 02:59:30.979841 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:30.979854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.980614 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:30.980617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:30.981208 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:30.981219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:30.981524 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:30.982894 (+ 1370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 02:59:30.982921 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0320 02:59:30.985624 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 02:59:30.990335 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:30.990345 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:30.990362 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:30.990935 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:30.991413 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 02:59:30.991418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 02:59:30.991420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0320 02:59:30.991629 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 02:59:30.993124 (+ 1495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 02:59:30.993131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 02:59:30.994081 (+ 950us) client_negotiation.cc:769] Sending connection context
0320 02:59:30.994405 (+ 324us) client_negotiation.cc:240] Negotiation successful
0320 02:59:30.994749 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":231,"threads_started":1}
I20260320 02:59:30.998478 3082 heartbeater.cc:344] Connected to a master server at 127.23.132.126:45613
I20260320 02:59:30.999092 3082 heartbeater.cc:461] Registering TS with master...
I20260320 02:59:30.999845 3082 heartbeater.cc:507] Master 127.23.132.126:45613 requested a full tablet report, sending...
I20260320 02:59:31.001725 2610 ts_manager.cc:194] Registered new tserver with Master: 901a49e207444ca29b9d932a2486f809 (127.23.132.67:34583)
I20260320 02:59:31.002630 2610 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.67@KRBTEST.COM'} at 127.23.132.67:37951
I20260320 02:59:31.013176 24081 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260320 02:59:31.026544 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.016288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.23.132.126:45613)
0320 02:59:31.016486 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.016490 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.016661 (+ 171us) server_negotiation.cc:408] Connection header received
0320 02:59:31.016936 (+ 275us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.016939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.017005 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.017114 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0320 02:59:31.018612 (+ 1498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.019379 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.020493 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.020719 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.022536 (+ 1817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 02:59:31.022564 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 02:59:31.022567 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 02:59:31.022601 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0320 02:59:31.024292 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:31.024998 (+ 706us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:31.025002 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:31.025004 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:31.025078 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 02:59:31.025322 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 02:59:31.025325 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 02:59:31.025383 (+ 58us) server_negotiation.cc:968] calling sasl_server_step()
0320 02:59:31.025676 (+ 293us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 02:59:31.025992 (+ 316us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.026220 (+ 228us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.026372 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260320 02:59:31.030345 2610 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260320 02:59:31.032752 2610 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260320 02:59:31.051440 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.046333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:37347 (local address 127.0.0.1:35028)
0320 02:59:31.047574 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.047588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:31.047702 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:31.048437 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:31.048441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:31.048465 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:31.048815 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.048823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.050296 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.050299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:31.051105 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.051113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.051204 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.051219 (+ 15us) client_negotiation.cc:769] Sending connection context
0320 02:59:31.051261 (+ 42us) client_negotiation.cc:240] Negotiation successful
0320 02:59:31.051310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"spinlock_wait_cycles":16256,"thread_start_us":93,"threads_started":1}
I20260320 02:59:31.051440 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.046252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:45353 (local address 127.0.0.1:58576)
0320 02:59:31.047444 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.047482 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:31.047622 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:31.048437 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:31.048441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:31.048468 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:31.048745 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.048755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.050081 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.050084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:31.050845 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.050853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.050966 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.051015 (+ 49us) client_negotiation.cc:769] Sending connection context
0320 02:59:31.051094 (+ 79us) client_negotiation.cc:240] Negotiation successful
0320 02:59:31.051195 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"spinlock_wait_cycles":4480,"thread_start_us":153,"threads_started":1}
I20260320 02:59:31.052379 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.047161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.23.132.67:34583)
0320 02:59:31.047493 (+ 332us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.047499 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.047796 (+ 297us) server_negotiation.cc:408] Connection header received
0320 02:59:31.047869 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.047873 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.048020 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.048139 (+ 119us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:31.048566 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.049634 (+ 1068us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.050967 (+ 1333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.051484 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.051980 (+ 496us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.052054 (+ 74us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.052152 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260320 02:59:31.052506 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.046440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.23.132.65:45353)
0320 02:59:31.047441 (+ 1001us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.047448 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.047873 (+ 425us) server_negotiation.cc:408] Connection header received
0320 02:59:31.048002 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.048008 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.048167 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.048291 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:31.048879 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.049955 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.051345 (+ 1390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.052012 (+ 667us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.052099 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.052156 (+ 57us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.052229 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":115,"threads_started":1}
I20260320 02:59:31.054204 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.046870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.23.132.66:37347)
0320 02:59:31.047922 (+ 1052us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.047928 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.047946 (+ 18us) server_negotiation.cc:408] Connection header received
0320 02:59:31.048016 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.048020 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.048167 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.048292 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:31.049206 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.050153 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.051688 (+ 1535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.053375 (+ 1687us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.053526 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.053611 (+ 85us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.053728 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":143,"threads_started":1}
I20260320 02:59:31.054613 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.047065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.67:34583 (local address 127.0.0.1:45910)
0320 02:59:31.047597 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.047610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:31.047702 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:31.048166 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:31.048172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:31.048194 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:31.048450 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.048457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.049814 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.049819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:31.050811 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.050821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.050934 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.051008 (+ 74us) client_negotiation.cc:769] Sending connection context
0320 02:59:31.051092 (+ 84us) client_negotiation.cc:240] Negotiation successful
0320 02:59:31.051195 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"spinlock_wait_cycles":12928,"thread_start_us":82,"threads_started":1}
I20260320 02:59:31.057454 2738 tablet_service.cc:1511] Processing CreateTablet for tablet 1ad80ff1daa046a58d92c95c79d58635 (DEFAULT_TABLE table=test-table [id=e2237eb894864b57b73915ce69fbbf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:31.058272 3015 tablet_service.cc:1511] Processing CreateTablet for tablet 1ad80ff1daa046a58d92c95c79d58635 (DEFAULT_TABLE table=test-table [id=e2237eb894864b57b73915ce69fbbf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:31.058167 2877 tablet_service.cc:1511] Processing CreateTablet for tablet 1ad80ff1daa046a58d92c95c79d58635 (DEFAULT_TABLE table=test-table [id=e2237eb894864b57b73915ce69fbbf5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260320 02:59:31.059217 3015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad80ff1daa046a58d92c95c79d58635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:31.059217 2877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad80ff1daa046a58d92c95c79d58635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:31.059295 2738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad80ff1daa046a58d92c95c79d58635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 02:59:31.068027 3101 tablet_bootstrap.cc:492] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Bootstrap starting.
I20260320 02:59:31.067216 3102 tablet_bootstrap.cc:492] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Bootstrap starting.
I20260320 02:59:31.068619 3103 tablet_bootstrap.cc:492] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: Bootstrap starting.
I20260320 02:59:31.071004 3103 tablet_bootstrap.cc:654] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:31.071473 3101 tablet_bootstrap.cc:654] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:31.071955 3103 log.cc:826] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:31.072615 3101 log.cc:826] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:31.074510 3102 tablet_bootstrap.cc:654] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Neither blocks nor log segments found. Creating new log.
I20260320 02:59:31.075476 3101 tablet_bootstrap.cc:492] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: No bootstrap required, opened a new log
I20260320 02:59:31.075934 3102 log.cc:826] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Log is configured to *not* fsync() on all Append() calls
I20260320 02:59:31.076049 3101 ts_tablet_manager.cc:1403] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260320 02:59:31.076247 3103 tablet_bootstrap.cc:492] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: No bootstrap required, opened a new log
I20260320 02:59:31.076449 3103 ts_tablet_manager.cc:1403] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260320 02:59:31.078817 3102 tablet_bootstrap.cc:492] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: No bootstrap required, opened a new log
I20260320 02:59:31.079120 3102 ts_tablet_manager.cc:1403] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260320 02:59:31.079448 3101 raft_consensus.cc:359] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.079728 3101 raft_consensus.cc:385] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:31.079792 3101 raft_consensus.cc:740] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11311dd8feb41cb8377ee5011f9b417, State: Initialized, Role: FOLLOWER
I20260320 02:59:31.080457 3101 consensus_queue.cc:260] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.080754 3103 raft_consensus.cc:359] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.081036 3103 raft_consensus.cc:385] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:31.081094 3103 raft_consensus.cc:740] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3afdc28ef074d02b62779a396597481, State: Initialized, Role: FOLLOWER
I20260320 02:59:31.081681 3103 consensus_queue.cc:260] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.081940 3101 ts_tablet_manager.cc:1434] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260320 02:59:31.082345 2945 heartbeater.cc:499] Master 127.23.132.126:45613 was elected leader, sending a full tablet report...
I20260320 02:59:31.082777 3103 ts_tablet_manager.cc:1434] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260320 02:59:31.083148 2806 heartbeater.cc:499] Master 127.23.132.126:45613 was elected leader, sending a full tablet report...
I20260320 02:59:31.086001 3102 raft_consensus.cc:359] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.086270 3102 raft_consensus.cc:385] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 02:59:31.086409 3102 raft_consensus.cc:740] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901a49e207444ca29b9d932a2486f809, State: Initialized, Role: FOLLOWER
I20260320 02:59:31.086889 3102 consensus_queue.cc:260] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.087554 3082 heartbeater.cc:499] Master 127.23.132.126:45613 was elected leader, sending a full tablet report...
I20260320 02:59:31.087882 3102 ts_tablet_manager.cc:1434] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260320 02:59:31.146862 2946 tablet.cc:2379] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:59:31.224418 3083 tablet.cc:2379] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260320 02:59:31.270089 2807 tablet.cc:2379] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260320 02:59:31.329824 3109 raft_consensus.cc:493] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260320 02:59:31.330076 3109 raft_consensus.cc:515] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.331213 3109 leader_election.cc:290] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3afdc28ef074d02b62779a396597481 (127.23.132.65:45353), c11311dd8feb41cb8377ee5011f9b417 (127.23.132.66:37347)
I20260320 02:59:31.335280 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.331515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:45353 (local address 127.23.132.67:48159)
0320 02:59:31.331625 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.331641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:31.331916 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:31.332349 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:31.332354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:31.332399 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:31.332729 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.332737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.333643 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.333647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:31.334801 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.334814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.335038 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.335060 (+ 22us) client_negotiation.cc:769] Sending connection context
0320 02:59:31.335105 (+ 45us) client_negotiation.cc:240] Negotiation successful
0320 02:59:31.335169 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260320 02:59:31.336025 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.331650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:48159 (local address 127.23.132.65:45353)
0320 02:59:31.331970 (+ 320us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.331975 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.331990 (+ 15us) server_negotiation.cc:408] Connection header received
0320 02:59:31.332058 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.332061 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.332111 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.332191 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:31.332871 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.333487 (+ 616us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.335055 (+ 1568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.335597 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.335658 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.335721 (+ 63us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.335793 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260320 02:59:31.336484 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.331515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:37347 (local address 127.23.132.67:54513)
0320 02:59:31.332555 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.332574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0320 02:59:31.332723 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 02:59:31.333163 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 02:59:31.333166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 02:59:31.333255 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0320 02:59:31.333650 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.333660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.334983 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.334987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 02:59:31.336059 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 02:59:31.336071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.336201 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.336221 (+ 20us) client_negotiation.cc:769] Sending connection context
0320 02:59:31.336269 (+ 48us) client_negotiation.cc:240] Negotiation successful
0320 02:59:31.336346 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":520,"thread_start_us":142,"threads_started":1}
I20260320 02:59:31.337309 2751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad80ff1daa046a58d92c95c79d58635" candidate_uuid: "901a49e207444ca29b9d932a2486f809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3afdc28ef074d02b62779a396597481" is_pre_election: true
I20260320 02:59:31.337607 2751 raft_consensus.cc:2468] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901a49e207444ca29b9d932a2486f809 in term 0.
I20260320 02:59:31.337670 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.331682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.67:54513 (local address 127.23.132.66:37347)
0320 02:59:31.331907 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.331911 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.332723 (+ 812us) server_negotiation.cc:408] Connection header received
0320 02:59:31.332934 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.332937 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.332990 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.333081 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0320 02:59:31.334170 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.334700 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.336285 (+ 1585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.337191 (+ 906us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.337238 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.337370 (+ 132us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.337485 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260320 02:59:31.338457 2968 leader_election.cc:304] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 901a49e207444ca29b9d932a2486f809, a3afdc28ef074d02b62779a396597481; no voters:
I20260320 02:59:31.338747 3109 raft_consensus.cc:2804] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260320 02:59:31.338833 3109 raft_consensus.cc:493] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260320 02:59:31.338874 3109 raft_consensus.cc:3060] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:31.339005 2898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad80ff1daa046a58d92c95c79d58635" candidate_uuid: "901a49e207444ca29b9d932a2486f809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11311dd8feb41cb8377ee5011f9b417" is_pre_election: true
I20260320 02:59:31.339417 2898 raft_consensus.cc:2468] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901a49e207444ca29b9d932a2486f809 in term 0.
I20260320 02:59:31.340512 3109 raft_consensus.cc:515] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.341081 3109 leader_election.cc:290] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [CANDIDATE]: Term 1 election: Requested vote from peers a3afdc28ef074d02b62779a396597481 (127.23.132.65:45353), c11311dd8feb41cb8377ee5011f9b417 (127.23.132.66:37347)
I20260320 02:59:31.341643 2751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad80ff1daa046a58d92c95c79d58635" candidate_uuid: "901a49e207444ca29b9d932a2486f809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3afdc28ef074d02b62779a396597481"
I20260320 02:59:31.341641 2898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad80ff1daa046a58d92c95c79d58635" candidate_uuid: "901a49e207444ca29b9d932a2486f809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11311dd8feb41cb8377ee5011f9b417"
I20260320 02:59:31.341956 2898 raft_consensus.cc:3060] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:31.341934 2751 raft_consensus.cc:3060] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 0 FOLLOWER]: Advancing to term 1
I20260320 02:59:31.343204 2898 raft_consensus.cc:2468] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901a49e207444ca29b9d932a2486f809 in term 1.
I20260320 02:59:31.343204 2751 raft_consensus.cc:2468] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901a49e207444ca29b9d932a2486f809 in term 1.
I20260320 02:59:31.343704 2971 leader_election.cc:304] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 901a49e207444ca29b9d932a2486f809, c11311dd8feb41cb8377ee5011f9b417; no voters:
I20260320 02:59:31.344020 3109 raft_consensus.cc:2804] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 02:59:31.344269 3109 raft_consensus.cc:697] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 1 LEADER]: Becoming Leader. State: Replica: 901a49e207444ca29b9d932a2486f809, State: Running, Role: LEADER
I20260320 02:59:31.344566 3109 consensus_queue.cc:237] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } }
I20260320 02:59:31.348349 2611 catalog_manager.cc:5671] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 reported cstate change: term changed from 0 to 1, leader changed from <none> to 901a49e207444ca29b9d932a2486f809 (127.23.132.67). New cstate: current_term: 1 leader_uuid: "901a49e207444ca29b9d932a2486f809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901a49e207444ca29b9d932a2486f809" member_type: VOTER last_known_addr { host: "127.23.132.67" port: 34583 } health_report { overall_health: HEALTHY } } }
I20260320 02:59:31.426978 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.423781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.23.132.67:34583)
0320 02:59:31.423958 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0320 02:59:31.423962 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0320 02:59:31.423980 (+ 18us) server_negotiation.cc:408] Connection header received
0320 02:59:31.424056 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 02:59:31.424058 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 02:59:31.424112 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 02:59:31.424193 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0320 02:59:31.424579 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.425081 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 02:59:31.425857 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 02:59:31.426018 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 02:59:31.426126 (+ 108us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0320 02:59:31.426552 (+ 426us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0320 02:59:31.426664 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0320 02:59:31.426810 (+ 146us) server_negotiation.cc:300] Negotiation successful
0320 02:59:31.426860 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260320 02:59:31.433743 2751 raft_consensus.cc:1275] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 1 FOLLOWER]: Refusing update from remote peer 901a49e207444ca29b9d932a2486f809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:31.434096 2898 raft_consensus.cc:1275] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 1 FOLLOWER]: Refusing update from remote peer 901a49e207444ca29b9d932a2486f809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260320 02:59:31.434598 3109 consensus_queue.cc:1048] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3afdc28ef074d02b62779a396597481" member_type: VOTER last_known_addr { host: "127.23.132.65" port: 45353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:31.434809 3111 consensus_queue.cc:1048] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11311dd8feb41cb8377ee5011f9b417" member_type: VOTER last_known_addr { host: "127.23.132.66" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260320 02:59:31.444799 3114 mvcc.cc:204] Tried to move back new op lower bound from 7266203940584247296 to 7266203940229812224. Current Snapshot: MvccSnapshot[applied={T|T < 7266203940584247296}]
I20260320 02:59:31.447228 3116 mvcc.cc:204] Tried to move back new op lower bound from 7266203940584247296 to 7266203940229812224. Current Snapshot: MvccSnapshot[applied={T|T < 7266203940584247296}]
I20260320 02:59:31.458595 2609 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:31.458972 2609 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260320 02:59:31.463920 2609 catalog_manager.cc:5958] T 00000000000000000000000000000000 P dd6721fda1564bbd85210f917554ad8d: Sending DeleteTablet for 3 replicas of tablet 1ad80ff1daa046a58d92c95c79d58635
I20260320 02:59:31.464933 2738 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ad80ff1daa046a58d92c95c79d58635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:31 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:58576
I20260320 02:59:31.465503 3015 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ad80ff1daa046a58d92c95c79d58635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:31 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:45910
I20260320 02:59:31.466085 2877 tablet_service.cc:1558] Processing DeleteTablet for tablet 1ad80ff1daa046a58d92c95c79d58635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-20 02:59:31 UTC) from {username='kudu', principal='kudu/127.23.132.126@KRBTEST.COM'} at 127.0.0.1:35028
I20260320 02:59:31.466950 3125 tablet_replica.cc:333] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: stopping tablet replica
I20260320 02:59:31.467741 3124 tablet_replica.cc:333] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481: stopping tablet replica
I20260320 02:59:31.468043 3124 raft_consensus.cc:2243] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:31.468415 3124 raft_consensus.cc:2272] T 1ad80ff1daa046a58d92c95c79d58635 P a3afdc28ef074d02b62779a396597481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:31.468827 3125 raft_consensus.cc:2243] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 1 LEADER]: Raft consensus shutting down.
I20260320 02:59:31.468863 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2675
I20260320 02:59:31.469712 3125 raft_consensus.cc:2272] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:31.473111 3125 ts_tablet_manager.cc:1916] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:31.473629 3126 tablet_replica.cc:333] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: stopping tablet replica
I20260320 02:59:31.474053 3126 raft_consensus.cc:2243] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260320 02:59:31.474573 3126 raft_consensus.cc:2272] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260320 02:59:31.479280 3125 ts_tablet_manager.cc:1929] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260320 02:59:31.479429 3125 log.cc:1199] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Deleting WAL directory at /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773975462196075-24081-0/minicluster-data/ts-2/wal/wals/1ad80ff1daa046a58d92c95c79d58635
I20260320 02:59:31.479811 3125 ts_tablet_manager.cc:1950] T 1ad80ff1daa046a58d92c95c79d58635 P 901a49e207444ca29b9d932a2486f809: Deleting consensus metadata
I20260320 02:59:31.481284 3126 ts_tablet_manager.cc:1916] T 1ad80ff1daa046a58d92c95c79d58635 P c11311dd8feb41cb8377ee5011f9b417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260320 02:59:31.482230 2597 catalog_manager.cc:5002] TS 901a49e207444ca29b9d932a2486f809 (127.23.132.67:34583): tablet 1ad80ff1daa046a58d92c95c79d58635 (table test-table [id=e2237eb894864b57b73915ce69fbbf5b]) successfully deleted
W20260320 02:59:31.482439 2595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.132.65:45353 (error 108)
I20260320 02:59:31.482753 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2813
I20260320 02:59:31.485518 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.485239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.65:45353 (local address 127.0.0.1:58580)
0320 02:59:31.485378 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.485437 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.65:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260320 02:59:31.485761 2595 catalog_manager.cc:4729] TS a3afdc28ef074d02b62779a396597481 (127.23.132.65:45353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ad80ff1daa046a58d92c95c79d58635: Network error: Client connection negotiation failed: client connection to 127.23.132.65:45353: connect: Connection refused (error 111)
I20260320 02:59:31.491756 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2952
I20260320 02:59:31.494211 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 02:59:31.492145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.66:37347 (local address 127.0.0.1:35030)
0320 02:59:31.493613 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0320 02:59:31.493689 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.66:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1266}
I20260320 02:59:31.502817 24081 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu with pid 2579
2026-03-20T02:59:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4282 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 20 02:59:31 dist-test-slave-v3hj krb5kdc[3130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 20 02:59:31 dist-test-slave-v3hj krb5kdc[3130](info): set up 2 sockets
Mar 20 02:59:31 dist-test-slave-v3hj krb5kdc[3130](info): commencing operation
krb5kdc: starting...
W20260320 02:59:33.596354 24081 mini_kdc.cc:121] Time spent starting KDC: real 2.069s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/test-user.keytab.
Mar 20 02:59:33 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-20T02:59:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-20T02:59:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.132.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.132.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/rangeradmin_127.23.132.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/rangeradmin_127.23.132.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.132.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.132.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/rangerlookup_127.23.132.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/rangerlookup_127.23.132.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.132.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/HTTP_127.23.132.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/HTTP_127.23.132.84@KRBTEST.COM.keytab.
I20260320 02:59:33.833495 24081 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknhX_zH/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/postgres -l logfile start
2026-03-20 02:59:38.352 UTC [3161] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-03-20 02:59:38.352 UTC [3161] LOG: listening on IPv4 address "127.23.132.84", port 44727
2026-03-20 02:59:38.363 UTC [3161] LOG: listening on Unix socket "/tmp/.s.PGSQL.44727"
2026-03-20 02:59:38.373 UTC [3166] LOG: database system was shut down at 2026-03-20 02:59:35 UTC
2026-03-20 02:59:38.380 UTC [3161] LOG: database system is ready to accept connections
I20260320 02:59:40.496240 24081 mini_postgres.cc:96] Postgres bound to 44727
2026-03-20 02:59:40.507 UTC [3172] FATAL: database "slave" does not exist
127.23.132.84:44727 - accepting connections
I20260320 02:59:40.508080 24081 mini_ranger.cc:161] Starting Ranger...
I20260320 02:59:40.541644 24081 mini_ranger.cc:84] Created miniranger Postgres user
I20260320 02:59:40.685497 24081 mini_ranger.cc:87] Created ranger Postgres database
I20260320 02:59:40.685606 24081 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-admin
2026-03-20 02:59:41,124 [I] DB FLAVOR :POSTGRES
2026-03-20 02:59:41,124 [I] --------- Verifying Ranger DB connection ---------
2026-03-20 02:59:41,125 [I] Checking connection..
2026-03-20 02:59:41,125 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-03-20 02:59:41,583 [I] Checking connection passed.
2026-03-20 02:59:41,584 [I] --------- Verifying version history table ---------
2026-03-20 02:59:41,584 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-20 02:59:42,115 [I] Table x_db_version_h does not exist in database ranger
2026-03-20 02:59:42,116 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-20 02:59:42,583 [I] Table x_db_version_h does not exist in database ranger
2026-03-20 02:59:42,583 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-03-20 02:59:42,584 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-03-20 02:59:43.004 UTC [3286] WARNING: there is no transaction in progress
2026-03-20 02:59:43,026 [I] create_dbversion_catalog.sql file imported successfully
2026-03-20 02:59:43,027 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-20 02:59:43,475 [I] Table x_db_version_h already exists in database 'ranger'
2026-03-20 02:59:43,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-03-20 02:59:43,475 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-03-20 02:59:43,918 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-03-20 02:59:44,351 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v3hj.c.gcp-upstream.internal','N') ;"
2026-03-20 02:59:44,814 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-03-20 02:59:45,264 [I] Table x_portal_user does not exist in database ranger
2026-03-20 02:59:45,265 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-03-20 02:59:45,767 [I] Table x_policy_ref_group does not exist in database ranger
2026-03-20 02:59:45,768 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-03-20 02:59:46,268 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-03-20 02:59:46,792 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-03-20 02:59:46,792 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-03-20 02:59:48.034 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.052 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.072 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.087 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.105 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.122 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.206 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.216 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.232 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.245 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.262 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.275 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.288 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.296 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.306 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:48.315 UTC [3486] WARNING: there is no transaction in progress
2026-03-20 02:59:49,066 [I] ranger_core_db_postgres.sql file imported successfully
2026-03-20 02:59:49,067 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-03-20 02:59:49,564 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-03-20 02:59:49,564 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-03-20 02:59:49,990 [I] Table x_portal_user already exists in database 'ranger'
2026-03-20 02:59:49,991 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-03-20 02:59:50,446 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-03-20 02:59:50,447 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-03-20 02:59:50,860 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-03-20 02:59:51,282 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v3hj.c.gcp-upstream.internal';"
2026-03-20 02:59:51,708 [I] CORE_DB_SCHEMA import status has been updated
2026-03-20 02:59:51,708 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-03-20 02:59:52,148 [I] DB_PATCHES have already been applied
I20260320 02:59:52.155002 24081 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-tasknhX_zH/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260320 02:59:52.155087 24081 mini_ranger.cc:193] Using host: 127.23.132.84
I20260320 02:59:52.158754 24081 mini_ranger.cc:239] Ranger admin URL: http://127.23.132.84:49091
Mar 20, 2026 2:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Mar 20, 2026 2:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Mar 20, 2026 2:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-admin/ews/webapp
Mar 20, 2026 2:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Mar 20, 2026 2:59:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-admin/ews/webapp] .....
Mar 20, 2026 2:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-admin/ews/webapp].
Mar 20, 2026 2:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260320 02:59:54.534904 24081 mini_ranger.cc:160] Time spent starting Ranger: real 14.027s user 0.000s sys 0.012s
Mar 20, 2026 3:00:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Mar 20, 2026 3:00:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Mar 20, 2026 3:00:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Mar 20, 2026 3:00:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Mar 20, 2026 3:00:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
I20260320 03:00:31.623672 24081 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.132.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.132.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/rangerkms_127.23.132.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.132.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/rangerkms_127.23.132.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.132.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.132.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/HTTP_127.23.132.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.132.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/HTTP_127.23.132.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Mar 20 03:00:31 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975631, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260320 03:00:31.786878 24081 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260320 03:00:31.802438 24081 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260320 03:00:31.972039 24081 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260320 03:00:31.972158 24081 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms
I20260320 03:00:31.972203 24081 mini_ranger_kms.cc:227] Using postgres at 127.23.132.84:44727
2026-03-20 03:00:33,086 [I] DB FLAVOR :POSTGRES
2026-03-20 03:00:33,087 [I] --------- Verifying Ranger DB connection ---------
2026-03-20 03:00:33,088 [I] Checking connection
2026-03-20 03:00:33,088 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-03-20 03:00:33,631 [I] connection success
2026-03-20 03:00:33,632 [I] --------- Verifying Ranger DB tables ---------
2026-03-20 03:00:33,632 [I] Verifying table ranger_masterkey in database rangerkms
2026-03-20 03:00:33,632 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-03-20 03:00:34,071 [I] Table ranger_masterkey does not exist in database rangerkms
2026-03-20 03:00:34,072 [I] --------- Importing Ranger Core DB Schema ---------
2026-03-20 03:00:34,072 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-03-20 03:00:34,072 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.132.84:44727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-03-20 03:00:34,576 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260320 03:00:35.527493 24081 mini_ranger_kms.cc:326] Created kms service
I20260320 03:00:35.866207 24081 mini_ranger_kms.cc:342] Created kudu user
I20260320 03:00:35.953193 24081 mini_ranger_kms.cc:359] Created rangerkms user
Mar 20 03:00:35 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975631, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.132.84@KRBTEST.COM
I20260320 03:00:36.603193 24081 mini_ranger_kms.cc:400] Added ranger policy
I20260320 03:00:36.603389 24081 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknhX_zH/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknhX_zH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260320 03:00:36.603410 24081 mini_ranger_kms.cc:240] Using host: 127.23.132.84
I20260320 03:00:36.609105 24081 mini_ranger_kms.cc:292] Ranger KMS PID: 3798
I20260320 03:00:36.609217 24081 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.23.132.84:38631
Mar 20, 2026 3:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Mar 20, 2026 3:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Mar 20, 2026 3:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Mar 20, 2026 3:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/ews/webapp] .....
Mar 20, 2026 3:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-kms/ews/webapp].
Mar 20, 2026 3:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Mar 20 03:00:40 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773975640, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.132.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Mar 20 03:00:41 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773975640, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.132.84@KRBTEST.COM for HTTP/127.23.132.84@KRBTEST.COM
I20260320 03:00:41.429000 24081 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 9.642s user 0.020s sys 0.360s
I20260320 03:00:41.429142 24081 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260320 03:00:41.429220 24081 mini_ranger_kms.cc:417] 127.23.132.84:38631/kms/v1/keys
Mar 20 03:00:41 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773975640, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.132.84@KRBTEST.COM for HTTP/127.23.132.84@KRBTEST.COM
Mar 20 03:00:45 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 03:00:45.236384 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:37715
--webserver_interface=127.23.132.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.132.126:37715
--ranger_config_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.132.84:38631/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 03:00:45.384860 3869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 03:00:45.385192 3869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 03:00:45.385257 3869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 03:00:45.390967 3869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260320 03:00:45.391069 3869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 03:00:45.391093 3869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 03:00:45.391111 3869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260320 03:00:45.391127 3869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260320 03:00:45.399677 3869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.132.84:38631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.132.126:37715
--ranger_config_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.132.126:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.132.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 03:00:45.400892 3869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 03:00:45.401881 3869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 03:00:45.414023 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 03:00:45.414189 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 03:00:45.414373 3869 server_base.cc:1061] running on GCE node
W20260320 03:00:45.414036 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 03:00:45.416344 3869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 03:00:45.417907 3869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 03:00:45.419158 3869 hybrid_clock.cc:648] HybridClock initialized: now 1773975645419150 us; error 30 us; skew 500 ppm
Mar 20 03:00:45 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975645, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 03:00:45.423061 3869 init.cc:377] Logged in from keytab as kudu/127.23.132.126@KRBTEST.COM (short username kudu)
I20260320 03:00:45.426993 3869 webserver.cc:492] Webserver started at http://127.23.132.126:35253/ using document root <none> and password file <none>
I20260320 03:00:45.427934 3869 fs_manager.cc:362] Metadata directory not provided
I20260320 03:00:45.427997 3869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 03:00:45.428217 3869 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Mar 20 03:00:45 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975645, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.126@KRBTEST.COM for HTTP/127.23.132.84@KRBTEST.COM
I20260320 03:00:45.519104 3869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data/instance:
uuid: "df1b88f3a8204aada25ad99ecfb79800"
format_stamp: "Formatted at 2026-03-20 03:00:45 on dist-test-slave-v3hj"
server_key: "877a749bbd756b80a4c61226f4c476e5"
server_key_iv: "d0bde2de52eac0721a42bf783ead4a05"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:45.520560 3869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal/instance:
uuid: "df1b88f3a8204aada25ad99ecfb79800"
format_stamp: "Formatted at 2026-03-20 03:00:45 on dist-test-slave-v3hj"
server_key: "877a749bbd756b80a4c61226f4c476e5"
server_key_iv: "d0bde2de52eac0721a42bf783ead4a05"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:45.533231 3869 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.013s sys 0.000s
I20260320 03:00:45.562790 3884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:45.566269 3869 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260320 03:00:45.566679 3869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal
uuid: "df1b88f3a8204aada25ad99ecfb79800"
format_stamp: "Formatted at 2026-03-20 03:00:45 on dist-test-slave-v3hj"
server_key: "877a749bbd756b80a4c61226f4c476e5"
server_key_iv: "d0bde2de52eac0721a42bf783ead4a05"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:45.567109 3869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 03:00:45.602708 3869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 03:00:45.616073 3869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 03:00:45.616539 3869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 03:00:45.634860 3938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.126:37715 every 8 connection(s)
I20260320 03:00:45.634939 3869 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.126:37715
I20260320 03:00:45.636834 3869 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/data/info.pb
I20260320 03:00:45.640518 3939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260320 03:00:45.642976 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 3869
I20260320 03:00:45.643446 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/wal/instance
I20260320 03:00:45.644085 24081 external_mini_cluster.cc:1468] Setting key ad505eb1975f41aa8eec380cdeee5ccf
I20260320 03:00:45.647552 3939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800: Bootstrap starting.
I20260320 03:00:45.651396 3939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800: Neither blocks nor log segments found. Creating new log.
I20260320 03:00:45.652155 3939 log.cc:826] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800: Log is configured to *not* fsync() on all Append() calls
I20260320 03:00:45.654533 3939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800: No bootstrap required, opened a new log
Mar 20 03:00:45 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 03:00:45.657752 3939 raft_consensus.cc:359] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1b88f3a8204aada25ad99ecfb79800" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 37715 } }
I20260320 03:00:45.658301 3939 raft_consensus.cc:385] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260320 03:00:45.658800 3939 raft_consensus.cc:740] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1b88f3a8204aada25ad99ecfb79800, State: Initialized, Role: FOLLOWER
I20260320 03:00:45.659591 3939 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1b88f3a8204aada25ad99ecfb79800" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 37715 } }
I20260320 03:00:45.659750 3939 raft_consensus.cc:399] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260320 03:00:45.659830 3939 raft_consensus.cc:493] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260320 03:00:45.660045 3939 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 0 FOLLOWER]: Advancing to term 1
I20260320 03:00:45.661019 3939 raft_consensus.cc:515] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1b88f3a8204aada25ad99ecfb79800" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 37715 } }
I20260320 03:00:45.661413 3939 leader_election.cc:304] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1b88f3a8204aada25ad99ecfb79800; no voters:
I20260320 03:00:45.661758 3939 leader_election.cc:290] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260320 03:00:45.662328 3944 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 1 FOLLOWER]: Leader election won for term 1
I20260320 03:00:45.663064 3939 sys_catalog.cc:565] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [sys.catalog]: configured and running, proceeding with master startup.
I20260320 03:00:45.663496 3944 raft_consensus.cc:697] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [term 1 LEADER]: Becoming Leader. State: Replica: df1b88f3a8204aada25ad99ecfb79800, State: Running, Role: LEADER
I20260320 03:00:45.663705 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:45.646014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.23.132.126:37715)
0320 03:00:45.646703 (+ 689us) server_negotiation.cc:207] Beginning negotiation
0320 03:00:45.646717 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0320 03:00:45.646771 (+ 54us) server_negotiation.cc:408] Connection header received
0320 03:00:45.648244 (+ 1473us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 03:00:45.648269 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 03:00:45.648572 (+ 303us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 03:00:45.648875 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0320 03:00:45.651121 (+ 2246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 03:00:45.652463 (+ 1342us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 03:00:45.653254 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 03:00:45.653511 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 03:00:45.656236 (+ 2725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 03:00:45.656260 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 03:00:45.656272 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 03:00:45.656358 (+ 86us) server_negotiation.cc:923] calling sasl_server_start()
0320 03:00:45.659429 (+ 3071us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 03:00:45.659955 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 03:00:45.659962 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 03:00:45.660028 (+ 66us) server_negotiation.cc:968] calling sasl_server_step()
0320 03:00:45.660127 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 03:00:45.660925 (+ 798us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 03:00:45.660929 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 03:00:45.660930 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0320 03:00:45.661418 (+ 488us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 03:00:45.661629 (+ 211us) server_negotiation.cc:1036] Waiting for connection context
0320 03:00:45.662524 (+ 895us) server_negotiation.cc:300] Negotiation successful
0320 03:00:45.662770 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260320 03:00:45.665917 3944 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1b88f3a8204aada25ad99ecfb79800" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 37715 } }
I20260320 03:00:45.670204 3946 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df1b88f3a8204aada25ad99ecfb79800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1b88f3a8204aada25ad99ecfb79800" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 37715 } } }
I20260320 03:00:45.671104 3939 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260320 03:00:45.671190 3946 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [sys.catalog]: This master's current role is: LEADER
I20260320 03:00:45.671167 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1b88f3a8204aada25ad99ecfb79800. Latest consensus state: current_term: 1 leader_uuid: "df1b88f3a8204aada25ad99ecfb79800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1b88f3a8204aada25ad99ecfb79800" member_type: VOTER last_known_addr { host: "127.23.132.126" port: 37715 } } }
I20260320 03:00:45.671569 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800 [sys.catalog]: This master's current role is: LEADER
I20260320 03:00:45.672013 3959 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260320 03:00:45.681200 3959 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260320 03:00:45.690539 3959 catalog_manager.cc:1357] Generated new cluster ID: d12fb2fb86b64964ae005814b68cc053
I20260320 03:00:45.690625 3959 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260320 03:00:45.714571 3959 catalog_manager.cc:1380] Generated new certificate authority record
I20260320 03:00:45.716365 3959 catalog_manager.cc:1514] Loading token signing keys...
I20260320 03:00:45.735690 3959 catalog_manager.cc:6044] T 00000000000000000000000000000000 P df1b88f3a8204aada25ad99ecfb79800: Generated new TSK 0
I20260320 03:00:45.737530 3959 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.23.132.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
I20260320 03:00:46.884078 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.132.84:38631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.65:0
--local_ip_for_outbound_sockets=127.23.132.65
--webserver_interface=127.23.132.65
--webserver_port=0
--tserver_master_addrs=127.23.132.126:37715
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260320 03:00:47.030478 3999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 03:00:47.030900 3999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 03:00:47.031039 3999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 03:00:47.036108 3999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 03:00:47.036278 3999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 03:00:47.036448 3999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.65
I20260320 03:00:47.042955 3999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.132.84:38631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.132.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:37715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.65
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 03:00:47.044872 3999 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 03:00:47.046124 3999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 03:00:47.054474 4006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 03:00:47.054474 4009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 03:00:47.055091 4007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 03:00:47.056257 3999 server_base.cc:1061] running on GCE node
I20260320 03:00:47.056740 3999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 03:00:47.057363 3999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 03:00:47.058530 3999 hybrid_clock.cc:648] HybridClock initialized: now 1773975647058516 us; error 32 us; skew 500 ppm
Mar 20 03:00:47 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 03:00:47.061867 3999 init.cc:377] Logged in from keytab as kudu/127.23.132.65@KRBTEST.COM (short username kudu)
I20260320 03:00:47.064987 3999 webserver.cc:492] Webserver started at http://127.23.132.65:34367/ using document root <none> and password file <none>
I20260320 03:00:47.066390 3999 fs_manager.cc:362] Metadata directory not provided
I20260320 03:00:47.066551 3999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 03:00:47.067286 3999 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Mar 20 03:00:47 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for HTTP/127.23.132.84@KRBTEST.COM
I20260320 03:00:47.099371 3999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data/instance:
uuid: "83e0637ae0484165b5c3bab5ba15ef48"
format_stamp: "Formatted at 2026-03-20 03:00:47 on dist-test-slave-v3hj"
server_key: "e5cdcc528dec8e2f0f92387e50fab1c6"
server_key_iv: "bf7e666a067fc9a58481a535dab8928c"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:47.100399 3999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance:
uuid: "83e0637ae0484165b5c3bab5ba15ef48"
format_stamp: "Formatted at 2026-03-20 03:00:47 on dist-test-slave-v3hj"
server_key: "e5cdcc528dec8e2f0f92387e50fab1c6"
server_key_iv: "bf7e666a067fc9a58481a535dab8928c"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:47.119498 3999 fs_manager.cc:696] Time spent creating directory manager: real 0.019s user 0.000s sys 0.004s
I20260320 03:00:47.155409 4019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:47.157296 3999 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260320 03:00:47.157740 3999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal
uuid: "83e0637ae0484165b5c3bab5ba15ef48"
format_stamp: "Formatted at 2026-03-20 03:00:47 on dist-test-slave-v3hj"
server_key: "e5cdcc528dec8e2f0f92387e50fab1c6"
server_key_iv: "bf7e666a067fc9a58481a535dab8928c"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:47.158239 3999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 03:00:47.186730 3999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 03:00:47.190873 3999 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 03:00:47.191318 3999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 03:00:47.192065 3999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 03:00:47.193308 3999 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 03:00:47.193418 3999 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:47.193539 3999 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 03:00:47.193614 3999 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:47.208446 3999 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.65:36275
I20260320 03:00:47.209862 3999 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/data/info.pb
I20260320 03:00:47.211336 4133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.65:36275 every 8 connection(s)
I20260320 03:00:47.218523 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 3999
I20260320 03:00:47.218680 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-0/wal/instance
I20260320 03:00:47.218977 24081 external_mini_cluster.cc:1468] Setting key cfe7e678a7c6a40525b812547ad09bec
Mar 20 03:00:47 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.65@KRBTEST.COM for kudu/127.23.132.126@KRBTEST.COM
I20260320 03:00:47.236270 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:47.213818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.132.65:56965 (local address 127.23.132.126:37715)
0320 03:00:47.219222 (+ 5404us) server_negotiation.cc:207] Beginning negotiation
0320 03:00:47.219231 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0320 03:00:47.219348 (+ 117us) server_negotiation.cc:408] Connection header received
0320 03:00:47.219958 (+ 610us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0320 03:00:47.219962 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0320 03:00:47.220041 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0320 03:00:47.220186 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
0320 03:00:47.222854 (+ 2668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 03:00:47.223743 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0320 03:00:47.224705 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0320 03:00:47.224914 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 03:00:47.228592 (+ 3678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0320 03:00:47.228626 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0320 03:00:47.228629 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0320 03:00:47.228666 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0320 03:00:47.230688 (+ 2022us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 03:00:47.233900 (+ 3212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 03:00:47.233906 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 03:00:47.233908 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0320 03:00:47.234000 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0320 03:00:47.234436 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0320 03:00:47.234440 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0320 03:00:47.234443 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0320 03:00:47.234690 (+ 247us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0320 03:00:47.234784 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0320 03:00:47.235568 (+ 784us) server_negotiation.cc:300] Negotiation successful
0320 03:00:47.235899 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5072,"thread_start_us":133,"threads_started":1}
I20260320 03:00:47.238476 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:47.215306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.65:56965)
0320 03:00:47.217090 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:47.217136 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0320 03:00:47.218558 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0320 03:00:47.220609 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0320 03:00:47.220627 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0320 03:00:47.221203 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0320 03:00:47.222532 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 03:00:47.222566 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 03:00:47.223939 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0320 03:00:47.223943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0320 03:00:47.224559 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0320 03:00:47.224570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0320 03:00:47.225233 (+ 663us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0320 03:00:47.226105 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0320 03:00:47.226151 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0320 03:00:47.228353 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0320 03:00:47.230858 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 03:00:47.230866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 03:00:47.230907 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0320 03:00:47.231278 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 03:00:47.234134 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0320 03:00:47.234138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0320 03:00:47.234141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0320 03:00:47.234318 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0320 03:00:47.234861 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0320 03:00:47.234867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0320 03:00:47.235231 (+ 364us) client_negotiation.cc:769] Sending connection context
0320 03:00:47.237136 (+ 1905us) client_negotiation.cc:240] Negotiation successful
0320 03:00:47.237499 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"thread_start_us":156,"threads_started":1}
I20260320 03:00:47.241396 4134 heartbeater.cc:344] Connected to a master server at 127.23.132.126:37715
I20260320 03:00:47.241683 4134 heartbeater.cc:461] Registering TS with master...
I20260320 03:00:47.242437 4134 heartbeater.cc:507] Master 127.23.132.126:37715 requested a full tablet report, sending...
I20260320 03:00:47.244365 3897 ts_manager.cc:194] Registered new tserver with Master: 83e0637ae0484165b5c3bab5ba15ef48 (127.23.132.65:36275)
WARNING: no policy specified for kudu/127.23.132.66@KRBTEST.COM; defaulting to no policy
I20260320 03:00:47.246922 3897 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.132.65@KRBTEST.COM'} at 127.23.132.65:56965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.132.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.23.132.126 from keytab /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:878)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:745)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.23.132.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.132.66@KRBTEST.COM" created.
I20260320 03:00:47.329321 3991 server.cc:273] Received an EOF from the subprocess
F20260320 03:00:47.330639 3963 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f39b46f8dcd google::LogMessage::Fail() at ??:0
@ 0x7f39b46fcb93 google::LogMessage::SendToLog() at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.23.132.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab.
@ 0x7f39b46f87cc google::LogMessage::Flush() at ??:0
I20260320 03:00:47.339579 24081 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
/tmp/dist-test-tasknhX_zH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.132.84:38631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.132.66:0
--local_ip_for_outbound_sockets=127.23.132.66
--webserver_interface=127.23.132.66
--webserver_port=0
--tserver_master_addrs=127.23.132.126:37715
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.23.132.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7f39b46f9f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f39b63b0157 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f39b63acf59 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f39b63b0e85 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f830bfc18a std::function<>::operator()() at ??:0
@ 0x7f39b5bf8214 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39b63e06db start_thread at ??:0
@ 0x7f39b38af61f clone at ??:0
W20260320 03:00:47.486021 4146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260320 03:00:47.486250 4146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260320 03:00:47.486310 4146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260320 03:00:47.490061 4146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260320 03:00:47.490141 4146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260320 03:00:47.490236 4146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.132.66
I20260320 03:00:47.494931 4146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.132.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.132.84:38631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/krb5kdc/kudu.keytab
--principal=kudu/127.23.132.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.132.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.132.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.23.132.126:37715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.132.66
--log_dir=/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a5468b651ad64ac6e09f94bae9fffde09c4211d
build type DEBUG
built by None at 20 Mar 2026 02:43:14 UTC on bdcb31816ec0
build id 11012
I20260320 03:00:47.496320 4146 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260320 03:00:47.497157 4146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260320 03:00:47.505335 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260320 03:00:47.505391 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 03:00:47.505663 4146 server_base.cc:1061] running on GCE node
W20260320 03:00:47.505391 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260320 03:00:47.506505 4146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260320 03:00:47.507500 4146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260320 03:00:47.508816 4146 hybrid_clock.cc:648] HybridClock initialized: now 1773975647508748 us; error 92 us; skew 500 ppm
Mar 20 03:00:47 dist-test-slave-v3hj krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260320 03:00:47.513087 4146 init.cc:377] Logged in from keytab as kudu/127.23.132.66@KRBTEST.COM (short username kudu)
I20260320 03:00:47.514369 4146 webserver.cc:492] Webserver started at http://127.23.132.66:39819/ using document root <none> and password file <none>
I20260320 03:00:47.515290 4146 fs_manager.cc:362] Metadata directory not provided
I20260320 03:00:47.515393 4146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260320 03:00:47.515643 4146 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Mar 20 03:00:47 dist-test-slave-v3hj krb5kdc[3130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773975647, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.132.66@KRBTEST.COM for HTTP/127.23.132.84@KRBTEST.COM
I20260320 03:00:47.532019 4146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data/instance:
uuid: "cde178cd1f2a443489ff0c966c76acc1"
format_stamp: "Formatted at 2026-03-20 03:00:47 on dist-test-slave-v3hj"
server_key: "015453963ba7cc3aa595380a6b6b8d9d"
server_key_iv: "32ddd0baafed85e578abad151f73a1b8"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:47.532603 4146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance:
uuid: "cde178cd1f2a443489ff0c966c76acc1"
format_stamp: "Formatted at 2026-03-20 03:00:47 on dist-test-slave-v3hj"
server_key: "015453963ba7cc3aa595380a6b6b8d9d"
server_key_iv: "32ddd0baafed85e578abad151f73a1b8"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:47.536911 4146 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260320 03:00:47.553828 4171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:47.563182 4146 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260320 03:00:47.563512 4146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal
uuid: "cde178cd1f2a443489ff0c966c76acc1"
format_stamp: "Formatted at 2026-03-20 03:00:47 on dist-test-slave-v3hj"
server_key: "015453963ba7cc3aa595380a6b6b8d9d"
server_key_iv: "32ddd0baafed85e578abad151f73a1b8"
server_key_version: "kuduclusterkey@0"
I20260320 03:00:47.563755 4146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260320 03:00:47.578706 4146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260320 03:00:47.583194 4146 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260320 03:00:47.583388 4146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260320 03:00:47.583961 4146 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260320 03:00:47.584909 4146 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260320 03:00:47.584959 4146 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:47.585011 4146 ts_tablet_manager.cc:616] Registered 0 tablets
I20260320 03:00:47.585040 4146 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260320 03:00:47.602744 4146 rpc_server.cc:307] RPC server started. Bound to: 127.23.132.66:41875
I20260320 03:00:47.604331 4146 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/data/info.pb
I20260320 03:00:47.602756 4290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.132.66:41875 every 8 connection(s)
I20260320 03:00:47.610649 24081 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasknhX_zH/build/debug/bin/kudu as pid 4146
I20260320 03:00:47.610792 24081 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasknhX_zH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773975462196075-24081-0/minicluster-data/ts-1/wal/instance
I20260320 03:00:47.610946 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:47.609100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:58359)
0320 03:00:47.609963 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:47.610099 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":272,"threads_started":1}
I20260320 03:00:47.611080 24081 external_mini_cluster.cc:1468] Setting key 2b7e79bc118de6108fbf12204141a7b7
W20260320 03:00:47.611968 4291 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:37715 (0 consecutive failures): Network error: Failed to ping master at 127.23.132.126:37715: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
I20260320 03:00:47.612593 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:47.612361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:59057)
0320 03:00:47.612484 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:47.612531 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260320 03:00:47.613232 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:47.613036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:56135)
0320 03:00:47.613138 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:47.613181 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260320 03:00:47.613417 4291 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260320 03:00:48.250912 4134 heartbeater.cc:499] Master 127.23.132.126:37715 was elected leader, sending a full tablet report...
I20260320 03:00:48.252113 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:48.251659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.65:52781)
0320 03:00:48.251931 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:48.251998 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
W20260320 03:00:48.252372 4134 heartbeater.cc:646] Failed to heartbeat to 127.23.132.126:37715 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
I20260320 03:00:48.614617 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:48.614108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:51719)
0320 03:00:48.614425 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:48.614489 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":76,"threads_started":1}
I20260320 03:00:49.253657 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:49.253085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.65:60467)
0320 03:00:49.253485 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:49.253551 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":96,"threads_started":1}
I20260320 03:00:49.616151 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:49.615583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:36737)
0320 03:00:49.615920 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:49.615984 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":82,"threads_started":1}
I20260320 03:00:50.255152 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:50.254645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.65:60493)
0320 03:00:50.254989 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:50.255050 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260320 03:00:50.617678 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:50.617207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:41855)
0320 03:00:50.617504 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:50.617570 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260320 03:00:51.256585 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:51.256100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.65:47351)
0320 03:00:51.256423 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:51.256484 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
I20260320 03:00:51.619127 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:51.618572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:47701)
0320 03:00:51.618962 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:51.619025 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":154,"threads_started":1}
I20260320 03:00:52.258131 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:52.257615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.65:36841)
0320 03:00:52.257969 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:52.258031 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":88,"threads_started":1}
I20260320 03:00:52.620849 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0320 03:00:52.620291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.132.126:37715 (local address 127.23.132.66:40507)
0320 03:00:52.620629 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0320 03:00:52.620690 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.132.126:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
F20260320 03:00:52.678825 24081 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1773975652 (unix time) try "date -d @1773975652" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005e11) received by PID 24081 (TID 0x7f42a32c5d80) from PID 24081; stack trace: ***
@ 0x7f42aef54980 (unknown) at ??:0
@ 0x7f42ac58de87 gsignal at ??:0
@ 0x7f42ac58f7f1 abort at ??:0
@ 0x7f42ad7fadcd google::LogMessage::Fail() at ??:0
@ 0x7f42ad7feb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f42ad7fa7cc google::LogMessage::Flush() at ??:0
@ 0x7f42ad7fbf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f42b058d43b kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f42b058a156 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55ecc5bda7bf kudu::SecurityITest::StartCluster() at ??:0
@ 0x55ecc5bc2622 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f42ae8020ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f42ae7f6bdb testing::Test::Run() at ??:0
@ 0x7f42ae7f6d9d testing::TestInfo::Run() at ??:0
@ 0x7f42ae7f7377 testing::TestSuite::Run() at ??:0
@ 0x7f42ae7f777c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f42ae80260d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f42ae7f6e63 testing::UnitTest::Run() at ??:0
@ 0x55ecc5c12b0e RUN_ALL_TESTS() at ??:0
@ 0x55ecc5c11cfb main at ??:0
@ 0x7f42ac570c87 __libc_start_main at ??:0
@ 0x55ecc5ba9efa _start at ??:0
2026-03-20 03:00:52.919 UTC [3691] FATAL: terminating connection due to unexpected postmaster exit
2026-03-20 03:00:52.919 UTC [3847] FATAL: terminating connection due to unexpected postmaster exit
2026-03-20 03:00:52.921 UTC [3686] FATAL: terminating connection due to unexpected postmaster exit
2026-03-20 03:00:52.922 UTC [3687] FATAL: terminating connection due to unexpected postmaster exit
2026-03-20 03:00:52.924 UTC [3688] FATAL: terminating connection due to unexpected postmaster exit
2026-03-20 03:00:52.926 UTC [3690] FATAL: terminating connection due to unexpected postmaster exit