[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 07:59:47 dist-test-slave-nhqz krb5kdc[12345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 07:59:47 dist-test-slave-nhqz krb5kdc[12345](info): set up 2 sockets
Apr 29 07:59:47 dist-test-slave-nhqz krb5kdc[12345](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260429 07:59:49.191713 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.957s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T07:59:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T07:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:49.330547 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:45079
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:49.429458 12361 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:49.429718 12361 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:49.429800 12361 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:49.432960 12361 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 07:59:49.433081 12361 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:49.433123 12361 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:49.433182 12361 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 07:59:49.433223 12361 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 07:59:49.437311 12361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:45079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:49.438354 12361 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:49.439217 12361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:49.444984 12366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:49.445253 12361 server_base.cc:1061] running on GCE node
W20260429 07:59:49.444984 12369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:49.445112 12367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:49.445946 12361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:49.446830 12361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:49.448035 12361 hybrid_clock.cc:648] HybridClock initialized: now 1777449589448009 us; error 43 us; skew 500 ppm
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:49.450757 12361 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 07:59:49.451802 12361 webserver.cc:492] Webserver started at http://127.12.12.190:39433/ using document root <none> and password file <none>
I20260429 07:59:49.452345 12361 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:49.452417 12361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:49.452615 12361 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:49.454339 12361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "8302350a4a774dbbb4febd5f6417b8c5"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "90542d3293ad3e3972bee20c45502ffe"
server_key_iv: "5cb3322eba14c6d14c911f787a599e4b"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.454833 12361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "8302350a4a774dbbb4febd5f6417b8c5"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "90542d3293ad3e3972bee20c45502ffe"
server_key_iv: "5cb3322eba14c6d14c911f787a599e4b"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.458196 12361 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260429 07:59:49.460284 12376 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.461225 12361 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:49.461373 12361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "8302350a4a774dbbb4febd5f6417b8c5"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "90542d3293ad3e3972bee20c45502ffe"
server_key_iv: "5cb3322eba14c6d14c911f787a599e4b"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.461477 12361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:49.472155 12361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:49.475121 12361 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:49.475322 12361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:49.482465 12361 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:45079
I20260429 07:59:49.482478 12428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:45079 every 8 connection(s)
I20260429 07:59:49.483459 12361 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 07:59:49.486011 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 12361
I20260429 07:59:49.486166 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 07:59:49.486480 12429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:49.487430 12338 external_mini_cluster.cc:1468] Setting key ba7e0718b98714135894c8266f7a05d4
I20260429 07:59:49.493091 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5: Bootstrap starting.
I20260429 07:59:49.495501 12429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:49.496309 12429 log.cc:826] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:49.498346 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5: No bootstrap required, opened a new log
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:49.501868 12429 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8302350a4a774dbbb4febd5f6417b8c5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45079 } }
I20260429 07:59:49.502117 12429 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:49.502204 12429 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8302350a4a774dbbb4febd5f6417b8c5, State: Initialized, Role: FOLLOWER
I20260429 07:59:49.502658 12429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8302350a4a774dbbb4febd5f6417b8c5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45079 } }
I20260429 07:59:49.502797 12429 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 07:59:49.502853 12429 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 07:59:49.503022 12429 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:49.503808 12429 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8302350a4a774dbbb4febd5f6417b8c5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45079 } }
I20260429 07:59:49.504307 12429 leader_election.cc:304] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8302350a4a774dbbb4febd5f6417b8c5; no voters:
I20260429 07:59:49.504691 12429 leader_election.cc:290] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 07:59:49.504791 12434 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:49.506084 12434 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8302350a4a774dbbb4febd5f6417b8c5, State: Running, Role: LEADER
I20260429 07:59:49.506392 12434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8302350a4a774dbbb4febd5f6417b8c5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45079 } }
I20260429 07:59:49.506892 12429 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 07:59:49.507773 12434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8302350a4a774dbbb4febd5f6417b8c5. Latest consensus state: current_term: 1 leader_uuid: "8302350a4a774dbbb4febd5f6417b8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8302350a4a774dbbb4febd5f6417b8c5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45079 } } }
I20260429 07:59:49.507884 12434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:49.508141 12435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8302350a4a774dbbb4febd5f6417b8c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8302350a4a774dbbb4febd5f6417b8c5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45079 } } }
I20260429 07:59:49.508256 12435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5 [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:49.508795 12432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:49.489988 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.12.12.190:45079)
0429 07:59:49.490493 (+ 505us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:49.490504 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:49.490843 (+ 339us) server_negotiation.cc:408] Connection header received
0429 07:59:49.493629 (+ 2786us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:49.493645 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:49.493900 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:49.494182 (+ 282us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:49.496390 (+ 2208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.497267 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:49.498099 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.498289 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:49.502291 (+ 4002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:49.502310 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:49.502321 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:49.502381 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:49.504374 (+ 1993us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:49.505124 (+ 750us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:49.505130 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:49.505134 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:49.505234 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:49.505606 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:49.505611 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:49.505613 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:49.506006 (+ 393us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:49.507500 (+ 1494us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:49.507655 (+ 155us) server_negotiation.cc:300] Negotiation successful
0429 07:59:49.507979 (+ 324us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":130,"threads_started":1}
W20260429 07:59:49.511678 12449 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 07:59:49.511762 12449 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 07:59:49.511842 12442 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 07:59:49.512531 12442 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 07:59:49.517622 12442 catalog_manager.cc:1357] Generated new cluster ID: 32f7c6c9901a46f994d37027bf2bfcf1
I20260429 07:59:49.517695 12442 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 07:59:49.529165 12442 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 07:59:49.529966 12442 catalog_manager.cc:1514] Loading token signing keys...
I20260429 07:59:49.538420 12442 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 8302350a4a774dbbb4febd5f6417b8c5: Generated new TSK 0
I20260429 07:59:49.539008 12442 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:49.589496 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45079
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:49.687750 12457 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:49.687984 12457 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:49.688107 12457 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:49.691354 12457 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:49.691455 12457 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:49.691555 12457 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 07:59:49.697582 12457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:45079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:49.698941 12457 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:49.699918 12457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:49.706102 12465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:49.706110 12463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:49.706115 12462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:49.706591 12457 server_base.cc:1061] running on GCE node
I20260429 07:59:49.706912 12457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:49.707439 12457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:49.708612 12457 hybrid_clock.cc:648] HybridClock initialized: now 1777449589708597 us; error 32 us; skew 500 ppm
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:49.711148 12457 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 07:59:49.712141 12457 webserver.cc:492] Webserver started at http://127.12.12.129:34751/ using document root <none> and password file <none>
I20260429 07:59:49.712661 12457 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:49.712709 12457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:49.712955 12457 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:49.714615 12457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "95895854859e40f19f465db415dc1811"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "1b1ca6269e1932842764f36a995cddcb"
server_key_iv: "aaf0c5caad82adb40df036438f18a4d2"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.715075 12457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "95895854859e40f19f465db415dc1811"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "1b1ca6269e1932842764f36a995cddcb"
server_key_iv: "aaf0c5caad82adb40df036438f18a4d2"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.718096 12457 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 07:59:49.720041 12472 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.721005 12457 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 07:59:49.721108 12457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "95895854859e40f19f465db415dc1811"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "1b1ca6269e1932842764f36a995cddcb"
server_key_iv: "aaf0c5caad82adb40df036438f18a4d2"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.721196 12457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:49.733644 12457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:49.735886 12457 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:49.736088 12457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:49.736662 12457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 07:59:49.737534 12457 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:49.737581 12457 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.737649 12457 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:49.737685 12457 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.746265 12457 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:33225
I20260429 07:59:49.746343 12585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:33225 every 8 connection(s)
I20260429 07:59:49.747190 12457 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 07:59:49.754884 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 12457
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:49.754989 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 07:59:49.755225 12338 external_mini_cluster.cc:1468] Setting key 31368c0cb43318ae0d4ed940b376f7e1
I20260429 07:59:49.759006 12432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:49.748862 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:52657 (local address 127.12.12.190:45079)
0429 07:59:49.749029 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:49.749033 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:49.749645 (+ 612us) server_negotiation.cc:408] Connection header received
0429 07:59:49.750478 (+ 833us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:49.750482 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:49.750539 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:49.750636 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:49.752107 (+ 1471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.752606 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:49.753259 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.753442 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:49.755631 (+ 2189us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:49.755648 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:49.755650 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:49.755674 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:49.757104 (+ 1430us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:49.757665 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:49.757667 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:49.757669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:49.757707 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:49.758005 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:49.758007 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:49.758008 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:49.758163 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:49.758281 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:49.758731 (+ 450us) server_negotiation.cc:300] Negotiation successful
0429 07:59:49.758856 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 07:59:49.759521 12588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:49.749124 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45079 (local address 127.12.12.129:52657)
0429 07:59:49.749511 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:49.749542 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:49.750286 (+ 744us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:49.750811 (+ 525us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:49.750818 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:49.751169 (+ 351us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:49.751955 (+ 786us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:49.751966 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.752727 (+ 761us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:49.752730 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:49.753141 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:49.753147 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.753344 (+ 197us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:49.753888 (+ 544us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:49.753905 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:49.755477 (+ 1572us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:49.757224 (+ 1747us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:49.757233 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:49.757246 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:49.757563 (+ 317us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:49.757787 (+ 224us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:49.757793 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:49.757798 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:49.757919 (+ 121us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:49.758286 (+ 367us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:49.758295 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:49.758496 (+ 201us) client_negotiation.cc:770] Sending connection context
0429 07:59:49.758682 (+ 186us) client_negotiation.cc:241] Negotiation successful
0429 07:59:49.758892 (+ 210us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
I20260429 07:59:49.760670 12586 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45079
I20260429 07:59:49.760937 12586 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:49.761535 12586 heartbeater.cc:507] Master 127.12.12.190:45079 requested a full tablet report, sending...
I20260429 07:59:49.763032 12393 ts_manager.cc:194] Registered new tserver with Master: 95895854859e40f19f465db415dc1811 (127.12.12.129:33225)
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 07:59:49.764183 12393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:52657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:49.801720 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45079
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:49.899186 12593 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:49.899439 12593 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:49.899533 12593 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:49.903384 12593 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:49.903491 12593 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:49.903602 12593 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 07:59:49.907734 12593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:45079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:49.908779 12593 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:49.909633 12593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:49.915760 12601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:49.915836 12598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:49.915881 12599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:49.916424 12593 server_base.cc:1061] running on GCE node
I20260429 07:59:49.916872 12593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:49.917398 12593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:49.918553 12593 hybrid_clock.cc:648] HybridClock initialized: now 1777449589918537 us; error 27 us; skew 500 ppm
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:49.921111 12593 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 07:59:49.922107 12593 webserver.cc:492] Webserver started at http://127.12.12.130:38513/ using document root <none> and password file <none>
I20260429 07:59:49.922600 12593 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:49.922644 12593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:49.922791 12593 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:49.924463 12593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "8319b27d185649588e9177d480cded84"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "0fac0183eb2be5dd68da601b598006c2"
server_key_iv: "647c17188106240cb0484308e02d5b86"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.924954 12593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "8319b27d185649588e9177d480cded84"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "0fac0183eb2be5dd68da601b598006c2"
server_key_iv: "647c17188106240cb0484308e02d5b86"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.928166 12593 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 07:59:49.930301 12608 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.931214 12593 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 07:59:49.931327 12593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "8319b27d185649588e9177d480cded84"
format_stamp: "Formatted at 2026-04-29 07:59:49 on dist-test-slave-nhqz"
server_key: "0fac0183eb2be5dd68da601b598006c2"
server_key_iv: "647c17188106240cb0484308e02d5b86"
server_key_version: "encryptionkey@0"
I20260429 07:59:49.931419 12593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:49.956687 12593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:49.959146 12593 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:49.959338 12593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:49.959892 12593 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 07:59:49.960736 12593 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:49.960806 12593 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.960903 12593 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:49.960949 12593 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:49.970336 12593 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:43147
I20260429 07:59:49.970366 12721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:43147 every 8 connection(s)
I20260429 07:59:49.971239 12593 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 07:59:49.977099 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 12593
I20260429 07:59:49.977210 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 07:59:49.977499 12338 external_mini_cluster.cc:1468] Setting key 25862ba9c101cff742f04a3173aa2ce8
Apr 29 07:59:49 dist-test-slave-nhqz krb5kdc[12345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449589, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:49.983448 12432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:49.972948 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:45849 (local address 127.12.12.190:45079)
0429 07:59:49.973104 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:49.973109 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:49.973782 (+ 673us) server_negotiation.cc:408] Connection header received
0429 07:59:49.974557 (+ 775us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:49.974560 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:49.974609 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:49.974684 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:49.976139 (+ 1455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.976629 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:49.977435 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.977638 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:49.979835 (+ 2197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:49.979858 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:49.979864 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:49.979900 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:49.981453 (+ 1553us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:49.982002 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:49.982009 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:49.982013 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:49.982067 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:49.982347 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:49.982353 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:49.982357 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:49.982564 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:49.982673 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:49.983115 (+ 442us) server_negotiation.cc:300] Negotiation successful
0429 07:59:49.983254 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 07:59:49.983897 12724 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:49.973260 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45079 (local address 127.12.12.130:45849)
0429 07:59:49.973646 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:49.973678 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:49.974388 (+ 710us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:49.974821 (+ 433us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:49.974829 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:49.975181 (+ 352us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:49.976013 (+ 832us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:49.976024 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.976729 (+ 705us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:49.976732 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:49.977314 (+ 582us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:49.977322 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:49.977479 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:49.978063 (+ 584us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:49.978081 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:49.979684 (+ 1603us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:49.981580 (+ 1896us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:49.981585 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:49.981595 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:49.981874 (+ 279us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:49.982171 (+ 297us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:49.982173 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:49.982175 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:49.982257 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:49.982667 (+ 410us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:49.982671 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:49.982860 (+ 189us) client_negotiation.cc:770] Sending connection context
0429 07:59:49.983035 (+ 175us) client_negotiation.cc:241] Negotiation successful
0429 07:59:49.983207 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
I20260429 07:59:49.985028 12722 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45079
I20260429 07:59:49.985293 12722 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:49.985875 12722 heartbeater.cc:507] Master 127.12.12.190:45079 requested a full tablet report, sending...
I20260429 07:59:49.987016 12393 ts_manager.cc:194] Registered new tserver with Master: 8319b27d185649588e9177d480cded84 (127.12.12.130:43147)
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 07:59:49.987799 12393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:45849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:50.027683 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45079
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:50.129802 12729 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:50.130007 12729 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:50.130123 12729 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:50.133586 12729 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:50.133664 12729 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:50.133791 12729 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 07:59:50.138175 12729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:45079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:50.139271 12729 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:50.140089 12729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:50.146492 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:50.146524 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:50.146589 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:50.146831 12729 server_base.cc:1061] running on GCE node
I20260429 07:59:50.147130 12729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:50.147631 12729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:50.148784 12729 hybrid_clock.cc:648] HybridClock initialized: now 1777449590148768 us; error 31 us; skew 500 ppm
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449590, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:50.151484 12729 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 07:59:50.152454 12729 webserver.cc:492] Webserver started at http://127.12.12.131:45835/ using document root <none> and password file <none>
I20260429 07:59:50.153017 12729 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:50.153065 12729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:50.153254 12729 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:50.155295 12729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "3abc55ef38424420a5bb0ef6d53b2cc5"
format_stamp: "Formatted at 2026-04-29 07:59:50 on dist-test-slave-nhqz"
server_key: "d3a2beffd0d81d71ea2983b68b5d4aa5"
server_key_iv: "eae93c4b4d61b46b62637b1bcea125ed"
server_key_version: "encryptionkey@0"
I20260429 07:59:50.155900 12729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "3abc55ef38424420a5bb0ef6d53b2cc5"
format_stamp: "Formatted at 2026-04-29 07:59:50 on dist-test-slave-nhqz"
server_key: "d3a2beffd0d81d71ea2983b68b5d4aa5"
server_key_iv: "eae93c4b4d61b46b62637b1bcea125ed"
server_key_version: "encryptionkey@0"
I20260429 07:59:50.160452 12729 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260429 07:59:50.162822 12744 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:50.164201 12729 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:50.164324 12729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "3abc55ef38424420a5bb0ef6d53b2cc5"
format_stamp: "Formatted at 2026-04-29 07:59:50 on dist-test-slave-nhqz"
server_key: "d3a2beffd0d81d71ea2983b68b5d4aa5"
server_key_iv: "eae93c4b4d61b46b62637b1bcea125ed"
server_key_version: "encryptionkey@0"
I20260429 07:59:50.164443 12729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:50.180498 12729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:50.183132 12729 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:50.183336 12729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:50.183897 12729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 07:59:50.184769 12729 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:50.184871 12729 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:50.184969 12729 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:50.185004 12729 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:50.194139 12729 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:36151
I20260429 07:59:50.194172 12857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:36151 every 8 connection(s)
I20260429 07:59:50.195065 12729 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449590, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:50.204133 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 12729
I20260429 07:59:50.204252 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 07:59:50.204556 12338 external_mini_cluster.cc:1468] Setting key f98894d5faf2375bc003a99ca177608f
I20260429 07:59:50.207345 12432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:50.196866 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:47489 (local address 127.12.12.190:45079)
0429 07:59:50.197036 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:50.197041 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:50.197703 (+ 662us) server_negotiation.cc:408] Connection header received
0429 07:59:50.198525 (+ 822us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:50.198528 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:50.198577 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:50.198652 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:50.200127 (+ 1475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.200660 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:50.201315 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.201451 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:50.203485 (+ 2034us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:50.203502 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:50.203505 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:50.203529 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:50.205174 (+ 1645us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:50.205748 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:50.205750 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:50.205752 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:50.205813 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:50.206290 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:50.206296 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:50.206298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:50.206480 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:50.206561 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:50.207112 (+ 551us) server_negotiation.cc:300] Negotiation successful
0429 07:59:50.207228 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 07:59:50.208099 12860 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:50.197171 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45079 (local address 127.12.12.131:47489)
0429 07:59:50.197566 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:50.197596 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:50.198352 (+ 756us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:50.198787 (+ 435us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:50.198794 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:50.199159 (+ 365us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:50.199975 (+ 816us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:50.199986 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.200761 (+ 775us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:50.200763 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:50.201218 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:50.201225 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.201371 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:50.201891 (+ 520us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:50.201907 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:50.203344 (+ 1437us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:50.205313 (+ 1969us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:50.205318 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:50.205326 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:50.205538 (+ 212us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:50.205948 (+ 410us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:50.205950 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:50.205952 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:50.206043 (+ 91us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:50.206586 (+ 543us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:50.206592 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:50.206845 (+ 253us) client_negotiation.cc:770] Sending connection context
0429 07:59:50.207064 (+ 219us) client_negotiation.cc:241] Negotiation successful
0429 07:59:50.207330 (+ 266us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260429 07:59:50.209224 12858 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45079
I20260429 07:59:50.209470 12858 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:50.209954 12858 heartbeater.cc:507] Master 127.12.12.190:45079 requested a full tablet report, sending...
I20260429 07:59:50.210912 12393 ts_manager.cc:194] Registered new tserver with Master: 3abc55ef38424420a5bb0ef6d53b2cc5 (127.12.12.131:36151)
I20260429 07:59:50.211450 12393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:47489
I20260429 07:59:50.219135 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449590, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 07:59:50.243335 12865 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:50.234906 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.12.12.129:33225)
0429 07:59:50.235309 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:50.235315 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:50.235336 (+ 21us) server_negotiation.cc:408] Connection header received
0429 07:59:50.235394 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:50.235399 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:50.235555 (+ 156us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:50.235692 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:50.236444 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.237258 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:50.237859 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.238020 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:50.240147 (+ 2127us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:50.240188 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:50.240199 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:50.240221 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:50.241871 (+ 1650us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:50.242241 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:50.242246 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:50.242247 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:50.242288 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:50.242509 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:50.242512 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:50.242513 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:50.242673 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:50.242808 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:50.242980 (+ 172us) server_negotiation.cc:300] Negotiation successful
0429 07:59:50.243136 (+ 156us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":136,"threads_started":1}
W20260429 07:59:50.244287 12500 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39114
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 07:59:50.268033 12865 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:50.260704 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.12.12.129:33225)
0429 07:59:50.260910 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:50.260914 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:50.261035 (+ 121us) server_negotiation.cc:408] Connection header received
0429 07:59:50.261146 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:50.261148 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:50.261197 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:50.261272 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:50.261996 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.262475 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:50.263091 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:50.263233 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:50.265326 (+ 2093us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:50.265345 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:50.265347 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:50.265380 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:50.266802 (+ 1422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:50.267121 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:50.267125 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:50.267126 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:50.267166 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:50.267381 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:50.267383 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:50.267384 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:50.267492 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:50.267575 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:50.267733 (+ 158us) server_negotiation.cc:300] Negotiation successful
0429 07:59:50.267816 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 07:59:50.269205 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 12457
I20260429 07:59:50.275210 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 12593
I20260429 07:59:50.284093 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 12729
I20260429 07:59:50.289731 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 12361
2026-04-29T07:59:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3083 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12872](info): set up 2 sockets
Apr 29 07:59:50 dist-test-slave-nhqz krb5kdc[12872](info): commencing operation
krb5kdc: starting...
W20260429 07:59:52.285713 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.973s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 07:59:52 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T07:59:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T07:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:52.420233 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:37183
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:52.520292 12888 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:52.520557 12888 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:52.520655 12888 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:52.523805 12888 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 07:59:52.523897 12888 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:52.523941 12888 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:52.524008 12888 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 07:59:52.524057 12888 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 07:59:52.528034 12888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:52.529075 12888 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:52.529898 12888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:52.535072 12893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:52.535171 12894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:52.535072 12896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:52.535367 12888 server_base.cc:1061] running on GCE node
I20260429 07:59:52.535840 12888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:52.536630 12888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:52.537803 12888 hybrid_clock.cc:648] HybridClock initialized: now 1777449592537801 us; error 57 us; skew 500 ppm
Apr 29 07:59:52 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449592, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:52.540254 12888 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 07:59:52.541257 12888 webserver.cc:492] Webserver started at http://127.12.12.190:39637/ using document root <none> and password file <none>
I20260429 07:59:52.541787 12888 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:52.541859 12888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:52.542047 12888 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:52.543576 12888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "a9b756e369ad45899932007b6f8baabe"
format_stamp: "Formatted at 2026-04-29 07:59:52 on dist-test-slave-nhqz"
server_key: "91de82b9af160a33659a288fa71e344d"
server_key_iv: "47beae39e927cb316d99343704d200a0"
server_key_version: "encryptionkey@0"
I20260429 07:59:52.544036 12888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "a9b756e369ad45899932007b6f8baabe"
format_stamp: "Formatted at 2026-04-29 07:59:52 on dist-test-slave-nhqz"
server_key: "91de82b9af160a33659a288fa71e344d"
server_key_iv: "47beae39e927cb316d99343704d200a0"
server_key_version: "encryptionkey@0"
I20260429 07:59:52.547263 12888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 07:59:52.549160 12903 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:52.550012 12888 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:52.550141 12888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "a9b756e369ad45899932007b6f8baabe"
format_stamp: "Formatted at 2026-04-29 07:59:52 on dist-test-slave-nhqz"
server_key: "91de82b9af160a33659a288fa71e344d"
server_key_iv: "47beae39e927cb316d99343704d200a0"
server_key_version: "encryptionkey@0"
I20260429 07:59:52.550236 12888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:52.571466 12888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:52.578207 12888 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:52.578439 12888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:52.585453 12955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:37183 every 8 connection(s)
I20260429 07:59:52.585464 12888 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:37183
I20260429 07:59:52.586535 12888 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 07:59:52.589215 12956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:52.594121 12956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe: Bootstrap starting.
I20260429 07:59:52.596127 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 12888
I20260429 07:59:52.596202 12956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:52.596314 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 07:59:52.596526 12338 external_mini_cluster.cc:1468] Setting key bbf4a893853c20194fb002a58d341e67
I20260429 07:59:52.597031 12956 log.cc:826] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:52.598805 12956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe: No bootstrap required, opened a new log
I20260429 07:59:52.601163 12956 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b756e369ad45899932007b6f8baabe" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37183 } }
I20260429 07:59:52.601352 12956 raft_consensus.cc:385] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:52.601433 12956 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b756e369ad45899932007b6f8baabe, State: Initialized, Role: FOLLOWER
I20260429 07:59:52.601925 12956 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b756e369ad45899932007b6f8baabe" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37183 } }
I20260429 07:59:52.602078 12956 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 07:59:52.602162 12956 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 07:59:52.602247 12956 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:52.603147 12956 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b756e369ad45899932007b6f8baabe" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37183 } }
Apr 29 07:59:52 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:52.603498 12956 leader_election.cc:304] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9b756e369ad45899932007b6f8baabe; no voters:
I20260429 07:59:52.603821 12956 leader_election.cc:290] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 07:59:52.603925 12961 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:52.604187 12961 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [term 1 LEADER]: Becoming Leader. State: Replica: a9b756e369ad45899932007b6f8baabe, State: Running, Role: LEADER
I20260429 07:59:52.604485 12961 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b756e369ad45899932007b6f8baabe" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37183 } }
I20260429 07:59:52.605103 12956 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [sys.catalog]: configured and running, proceeding with master startup.
I20260429 07:59:52.606174 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9b756e369ad45899932007b6f8baabe. Latest consensus state: current_term: 1 leader_uuid: "a9b756e369ad45899932007b6f8baabe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b756e369ad45899932007b6f8baabe" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37183 } } }
I20260429 07:59:52.606313 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:52.606185 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9b756e369ad45899932007b6f8baabe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9b756e369ad45899932007b6f8baabe" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37183 } } }
I20260429 07:59:52.606383 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:52.607403 12970 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 07:59:52.610257 12970 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 07:59:52.610941 12960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:52.597619 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.12.12.190:37183)
0429 07:59:52.598001 (+ 382us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:52.598010 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:52.598040 (+ 30us) server_negotiation.cc:408] Connection header received
0429 07:59:52.598558 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:52.598574 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:52.598870 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:52.599178 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:52.600044 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:52.600778 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:52.601486 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:52.601743 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:52.604196 (+ 2453us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:52.604215 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:52.604227 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:52.604254 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:52.608373 (+ 4119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:52.608857 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:52.608865 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:52.608873 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:52.608966 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:52.609242 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:52.609245 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:52.609247 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:52.609530 (+ 283us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:52.609705 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:52.610064 (+ 359us) server_negotiation.cc:300] Negotiation successful
0429 07:59:52.610449 (+ 385us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":118,"threads_started":1}
I20260429 07:59:52.615360 12970 catalog_manager.cc:1357] Generated new cluster ID: cf17f2e48d30456692c952a8e30856c2
I20260429 07:59:52.615425 12970 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 07:59:52.621301 12970 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 07:59:52.622012 12970 catalog_manager.cc:1514] Loading token signing keys...
I20260429 07:59:52.629766 12970 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a9b756e369ad45899932007b6f8baabe: Generated new TSK 0
I20260429 07:59:52.630347 12970 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:52.681905 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:37183
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:52.796710 12984 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:52.796924 12984 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:52.797024 12984 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:52.800246 12984 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:52.800308 12984 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:52.800449 12984 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 07:59:52.804625 12984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:37183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:52.805655 12984 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:52.806618 12984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:52.812913 12990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:52.812907 12989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:52.812992 12984 server_base.cc:1061] running on GCE node
W20260429 07:59:52.812907 12992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:52.813512 12984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:52.814013 12984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:52.815186 12984 hybrid_clock.cc:648] HybridClock initialized: now 1777449592815161 us; error 34 us; skew 500 ppm
Apr 29 07:59:52 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449592, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:52.818086 12984 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 07:59:52.819011 12984 webserver.cc:492] Webserver started at http://127.12.12.129:46143/ using document root <none> and password file <none>
I20260429 07:59:52.819523 12984 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:52.819568 12984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:52.819717 12984 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:52.821403 12984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "9ca352c1be5343f0bd73f7495fb17e11"
format_stamp: "Formatted at 2026-04-29 07:59:52 on dist-test-slave-nhqz"
server_key: "f4191fec55e5f88dabe1fbee29696b6e"
server_key_iv: "d91127bd36d44370f2dcdc5f8642d31e"
server_key_version: "encryptionkey@0"
I20260429 07:59:52.821826 12984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "9ca352c1be5343f0bd73f7495fb17e11"
format_stamp: "Formatted at 2026-04-29 07:59:52 on dist-test-slave-nhqz"
server_key: "f4191fec55e5f88dabe1fbee29696b6e"
server_key_iv: "d91127bd36d44370f2dcdc5f8642d31e"
server_key_version: "encryptionkey@0"
I20260429 07:59:52.825030 12984 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 07:59:52.827040 12999 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:52.827968 12984 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:52.828114 12984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "9ca352c1be5343f0bd73f7495fb17e11"
format_stamp: "Formatted at 2026-04-29 07:59:52 on dist-test-slave-nhqz"
server_key: "f4191fec55e5f88dabe1fbee29696b6e"
server_key_iv: "d91127bd36d44370f2dcdc5f8642d31e"
server_key_version: "encryptionkey@0"
I20260429 07:59:52.828215 12984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:52.846443 12984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:52.849406 12984 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:52.849627 12984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:52.850224 12984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 07:59:52.851140 12984 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:52.851212 12984 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:52.851289 12984 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:52.851342 12984 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:52.861189 12984 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:39333
I20260429 07:59:52.861246 13112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:39333 every 8 connection(s)
I20260429 07:59:52.862142 12984 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 07:59:52.867964 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 12984
I20260429 07:59:52.868070 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 07:59:52.868300 12338 external_mini_cluster.cc:1468] Setting key de3335c67fcfd2a781cbd1c403434144
Apr 29 07:59:52 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449592, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:52.874691 12960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:52.863882 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:50247 (local address 127.12.12.190:37183)
0429 07:59:52.864032 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:52.864035 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:52.864721 (+ 686us) server_negotiation.cc:408] Connection header received
0429 07:59:52.865568 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:52.865571 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:52.865641 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:52.865728 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:52.867149 (+ 1421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:52.867651 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:52.868476 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:52.868619 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:52.871204 (+ 2585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:52.871227 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:52.871230 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:52.871255 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:52.872735 (+ 1480us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:52.873261 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:52.873265 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:52.873267 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:52.873327 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:52.873617 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:52.873622 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:52.873624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:52.873772 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:52.873875 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:52.874355 (+ 480us) server_negotiation.cc:300] Negotiation successful
0429 07:59:52.874493 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 07:59:52.875319 13115 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:52.864177 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37183 (local address 127.12.12.129:50247)
0429 07:59:52.864582 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:52.864613 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:52.865363 (+ 750us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:52.865862 (+ 499us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:52.865870 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:52.866217 (+ 347us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:52.866999 (+ 782us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:52.867010 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:52.867790 (+ 780us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:52.867795 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:52.868336 (+ 541us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:52.868347 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:52.868522 (+ 175us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:52.869131 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:52.869155 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:52.871048 (+ 1893us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:52.872894 (+ 1846us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:52.872900 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:52.872911 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:52.873152 (+ 241us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:52.873420 (+ 268us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:52.873425 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:52.873427 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:52.873520 (+ 93us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:52.873886 (+ 366us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:52.873893 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:52.874116 (+ 223us) client_negotiation.cc:770] Sending connection context
0429 07:59:52.874316 (+ 200us) client_negotiation.cc:241] Negotiation successful
0429 07:59:52.874527 (+ 211us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 07:59:52.876605 13113 heartbeater.cc:344] Connected to a master server at 127.12.12.190:37183
I20260429 07:59:52.876933 13113 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:52.877458 13113 heartbeater.cc:507] Master 127.12.12.190:37183 requested a full tablet report, sending...
I20260429 07:59:52.878934 12920 ts_manager.cc:194] Registered new tserver with Master: 9ca352c1be5343f0bd73f7495fb17e11 (127.12.12.129:39333)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
I20260429 07:59:52.880251 12920 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:50247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:52.912532 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:37183
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:53.011282 13120 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:53.011583 13120 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:53.011706 13120 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:53.015288 13120 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:53.015350 13120 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:53.015492 13120 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 07:59:53.019711 13120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:37183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:53.020744 13120 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:53.021613 13120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:53.027712 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:53.027763 13126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:53.027709 13125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:53.027900 13120 server_base.cc:1061] running on GCE node
I20260429 07:59:53.028240 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:53.028697 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:53.029843 13120 hybrid_clock.cc:648] HybridClock initialized: now 1777449593029816 us; error 39 us; skew 500 ppm
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:53.032382 13120 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 07:59:53.033557 13120 webserver.cc:492] Webserver started at http://127.12.12.130:39697/ using document root <none> and password file <none>
I20260429 07:59:53.034063 13120 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:53.034129 13120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:53.034330 13120 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:53.035914 13120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "dc7e6522ec8647aebcb267b765ef3dc9"
format_stamp: "Formatted at 2026-04-29 07:59:53 on dist-test-slave-nhqz"
server_key: "08d68a9c40d39e5202247c1b22fad898"
server_key_iv: "12bd3910568f53a4109bd1c3874b39c1"
server_key_version: "encryptionkey@0"
I20260429 07:59:53.036509 13120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "dc7e6522ec8647aebcb267b765ef3dc9"
format_stamp: "Formatted at 2026-04-29 07:59:53 on dist-test-slave-nhqz"
server_key: "08d68a9c40d39e5202247c1b22fad898"
server_key_iv: "12bd3910568f53a4109bd1c3874b39c1"
server_key_version: "encryptionkey@0"
I20260429 07:59:53.039835 13120 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 07:59:53.042047 13135 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:53.043056 13120 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 07:59:53.043192 13120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "dc7e6522ec8647aebcb267b765ef3dc9"
format_stamp: "Formatted at 2026-04-29 07:59:53 on dist-test-slave-nhqz"
server_key: "08d68a9c40d39e5202247c1b22fad898"
server_key_iv: "12bd3910568f53a4109bd1c3874b39c1"
server_key_version: "encryptionkey@0"
I20260429 07:59:53.043291 13120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:53.056716 13120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:53.059022 13120 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:53.059212 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:53.059715 13120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 07:59:53.060534 13120 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:53.060601 13120 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:53.060668 13120 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:53.060717 13120 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:53.069136 13120 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:35679
I20260429 07:59:53.069178 13248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:35679 every 8 connection(s)
I20260429 07:59:53.069991 13120 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:53.078213 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 13120
I20260429 07:59:53.078321 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 07:59:53.078580 12338 external_mini_cluster.cc:1468] Setting key 22fca0b66af9b478280e563108d0f2b2
I20260429 07:59:53.081334 12960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.071551 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:48689 (local address 127.12.12.190:37183)
0429 07:59:53.071696 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.071701 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.072283 (+ 582us) server_negotiation.cc:408] Connection header received
0429 07:59:53.073052 (+ 769us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.073055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.073101 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.073172 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:53.074608 (+ 1436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.075109 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.075664 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.075785 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.077798 (+ 2013us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:53.077816 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:53.077818 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:53.077840 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:53.079444 (+ 1604us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.079973 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.079978 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.079979 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.080032 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.080324 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.080328 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.080330 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.080492 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:53.080597 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.081069 (+ 472us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.081180 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 07:59:53.081933 13251 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.071811 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37183 (local address 127.12.12.130:48689)
0429 07:59:53.072161 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.072190 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.072897 (+ 707us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.073299 (+ 402us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.073305 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.073688 (+ 383us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:53.074486 (+ 798us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.074498 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.075203 (+ 705us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.075205 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.075576 (+ 371us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.075582 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.075722 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.076258 (+ 536us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:53.076274 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:53.077675 (+ 1401us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:53.079565 (+ 1890us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:53.079571 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:53.079582 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:53.079842 (+ 260us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:53.080125 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:53.080128 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:53.080129 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:53.080222 (+ 93us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:53.080596 (+ 374us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:53.080602 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:53.080791 (+ 189us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.081022 (+ 231us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.081237 (+ 215us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260429 07:59:53.082870 13249 heartbeater.cc:344] Connected to a master server at 127.12.12.190:37183
I20260429 07:59:53.083135 13249 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:53.083722 13249 heartbeater.cc:507] Master 127.12.12.190:37183 requested a full tablet report, sending...
I20260429 07:59:53.084858 12920 ts_manager.cc:194] Registered new tserver with Master: dc7e6522ec8647aebcb267b765ef3dc9 (127.12.12.130:35679)
I20260429 07:59:53.085424 12920 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:48689
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:53.121973 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:37183
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:53.221385 13256 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:53.221633 13256 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:53.221724 13256 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:53.224987 13256 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:53.225080 13256 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:53.225191 13256 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 07:59:53.229378 13256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:37183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:53.230515 13256 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:53.231359 13256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:53.237787 13262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:53.237802 13261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:53.237814 13264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:53.238188 13256 server_base.cc:1061] running on GCE node
I20260429 07:59:53.238559 13256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:53.239099 13256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:53.240267 13256 hybrid_clock.cc:648] HybridClock initialized: now 1777449593240253 us; error 37 us; skew 500 ppm
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:53.242867 13256 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 07:59:53.243850 13256 webserver.cc:492] Webserver started at http://127.12.12.131:44567/ using document root <none> and password file <none>
I20260429 07:59:53.244340 13256 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:53.244402 13256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:53.244553 13256 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:53.246168 13256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "b4320be2158148dc9db6b266406eace0"
format_stamp: "Formatted at 2026-04-29 07:59:53 on dist-test-slave-nhqz"
server_key: "549cbda15ed476e50c42a6eda382d0a6"
server_key_iv: "010908e2efa1f0e00d98a42564ce70dd"
server_key_version: "encryptionkey@0"
I20260429 07:59:53.246678 13256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "b4320be2158148dc9db6b266406eace0"
format_stamp: "Formatted at 2026-04-29 07:59:53 on dist-test-slave-nhqz"
server_key: "549cbda15ed476e50c42a6eda382d0a6"
server_key_iv: "010908e2efa1f0e00d98a42564ce70dd"
server_key_version: "encryptionkey@0"
I20260429 07:59:53.249840 13256 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 07:59:53.251864 13271 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:53.252734 13256 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:53.252877 13256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "b4320be2158148dc9db6b266406eace0"
format_stamp: "Formatted at 2026-04-29 07:59:53 on dist-test-slave-nhqz"
server_key: "549cbda15ed476e50c42a6eda382d0a6"
server_key_iv: "010908e2efa1f0e00d98a42564ce70dd"
server_key_version: "encryptionkey@0"
I20260429 07:59:53.252971 13256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:53.274358 13256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:53.277241 13256 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:53.277467 13256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:53.278034 13256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 07:59:53.278901 13256 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:53.278949 13256 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:53.279018 13256 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:53.279050 13256 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:53.288623 13256 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:40961
I20260429 07:59:53.288645 13384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:40961 every 8 connection(s)
I20260429 07:59:53.289630 13256 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:53.297574 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 13256
I20260429 07:59:53.297713 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 07:59:53.297938 12338 external_mini_cluster.cc:1468] Setting key 7eb6978b74fe5ccf26688cc789a8fa8c
I20260429 07:59:53.301649 12960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.291389 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:56131 (local address 127.12.12.190:37183)
0429 07:59:53.291537 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.291541 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.292173 (+ 632us) server_negotiation.cc:408] Connection header received
0429 07:59:53.293064 (+ 891us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.293068 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.293128 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.293243 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:53.294705 (+ 1462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.295196 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.295829 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.295996 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.298278 (+ 2282us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:53.298304 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:53.298306 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:53.298331 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:53.299827 (+ 1496us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.300431 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.300433 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.300435 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.300479 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.300720 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.300722 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.300723 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.300923 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:53.301000 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.301442 (+ 442us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.301532 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 07:59:53.302309 13387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.291647 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37183 (local address 127.12.12.131:56131)
0429 07:59:53.292031 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.292062 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.292875 (+ 813us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.293404 (+ 529us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.293412 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.293756 (+ 344us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:53.294555 (+ 799us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.294567 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.295321 (+ 754us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.295326 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.295711 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.295717 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.295923 (+ 206us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.296530 (+ 607us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:53.296548 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:53.298138 (+ 1590us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:53.300073 (+ 1935us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:53.300080 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:53.300092 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:53.300338 (+ 246us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:53.300564 (+ 226us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:53.300566 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:53.300567 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:53.300655 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:53.301014 (+ 359us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:53.301018 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:53.301231 (+ 213us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.301397 (+ 166us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.301576 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260429 07:59:53.303369 13385 heartbeater.cc:344] Connected to a master server at 127.12.12.190:37183
I20260429 07:59:53.303607 13385 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:53.304082 13385 heartbeater.cc:507] Master 127.12.12.190:37183 requested a full tablet report, sending...
I20260429 07:59:53.305076 12920 ts_manager.cc:194] Registered new tserver with Master: b4320be2158148dc9db6b266406eace0 (127.12.12.131:40961)
I20260429 07:59:53.305639 12920 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:56131
I20260429 07:59:53.311913 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 07:59:53.321828 12960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.315026 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.12.12.190:37183)
0429 07:59:53.315194 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.315198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.315311 (+ 113us) server_negotiation.cc:408] Connection header received
0429 07:59:53.315466 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.315469 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.315519 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.315622 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:53.316404 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.316975 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.317634 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.317837 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.318809 (+ 972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:53.318825 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:53.318828 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:53.318857 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:53.319979 (+ 1122us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.320545 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.320551 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.320553 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.320613 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.320923 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.320927 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.320929 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.321124 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:53.321285 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.321520 (+ 235us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.321637 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260429 07:59:53.325829 12920 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 07:59:53.328132 12920 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 07:59:53.340529 13398 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.336685 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:39333 (local address 127.0.0.1:53840)
0429 07:59:53.337402 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.337416 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.337493 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.337989 (+ 496us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.337992 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.338050 (+ 58us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:53.338242 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.338249 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.339147 (+ 898us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.339152 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.340037 (+ 885us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.340045 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.340145 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.340191 (+ 46us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.340279 (+ 88us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.340350 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"spinlock_wait_cycles":40192,"thread_start_us":134,"threads_started":1}
I20260429 07:59:53.341545 13397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.336133 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.12.12.130:35679)
0429 07:59:53.337085 (+ 952us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.337092 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.337114 (+ 22us) server_negotiation.cc:408] Connection header received
0429 07:59:53.337203 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.337208 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.337404 (+ 196us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.338085 (+ 681us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:53.338211 (+ 126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.339320 (+ 1109us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.340599 (+ 1279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.341113 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.341234 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.341307 (+ 73us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.341404 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":100,"threads_started":1}
I20260429 07:59:53.341558 13400 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.336573 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.12.12.131:40961)
0429 07:59:53.337446 (+ 873us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.337453 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.337476 (+ 23us) server_negotiation.cc:408] Connection header received
0429 07:59:53.337552 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.337557 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.337693 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.337850 (+ 157us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:53.338391 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.339475 (+ 1084us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.340868 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.341236 (+ 368us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.341307 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.341388 (+ 81us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.341453 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":348,"threads_started":1}
I20260429 07:59:53.341778 13401 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.336907 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.12.12.129:39333)
0429 07:59:53.337272 (+ 365us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.337277 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.337581 (+ 304us) server_negotiation.cc:408] Connection header received
0429 07:59:53.337671 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.337676 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.337857 (+ 181us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.338002 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:53.338356 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.339040 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.340902 (+ 1862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.341403 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.341482 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.341542 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.341626 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":102,"threads_started":1}
I20260429 07:59:53.340530 13396 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.336021 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:35679 (local address 127.0.0.1:37942)
0429 07:59:53.336328 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.336360 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.336483 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.337578 (+ 1095us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.337584 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.337608 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:53.337932 (+ 324us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.337942 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.339431 (+ 1489us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.339434 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.340222 (+ 788us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.340229 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.340317 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.340329 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.340373 (+ 44us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.340414 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260429 07:59:53.340747 13399 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.336422 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:40961 (local address 127.0.0.1:47332)
0429 07:59:53.336997 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.337012 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.337097 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.337984 (+ 887us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.337991 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.338045 (+ 54us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:53.338261 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.338268 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.339591 (+ 1323us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.339595 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.340367 (+ 772us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.340377 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.340503 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.340523 (+ 20us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.340592 (+ 69us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.340650 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"spinlock_wait_cycles":9728,"thread_start_us":95,"threads_started":1}
I20260429 07:59:53.344892 13183 tablet_service.cc:1511] Processing CreateTablet for tablet fe70b1e39f6c4a8ca860b49760822bcf (DEFAULT_TABLE table=test-table [id=477814fafebd406db1614e0483425146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:53.344892 13047 tablet_service.cc:1511] Processing CreateTablet for tablet fe70b1e39f6c4a8ca860b49760822bcf (DEFAULT_TABLE table=test-table [id=477814fafebd406db1614e0483425146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:53.345868 13047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe70b1e39f6c4a8ca860b49760822bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:53.345867 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe70b1e39f6c4a8ca860b49760822bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:53.350230 13319 tablet_service.cc:1511] Processing CreateTablet for tablet fe70b1e39f6c4a8ca860b49760822bcf (DEFAULT_TABLE table=test-table [id=477814fafebd406db1614e0483425146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:53.351052 13319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe70b1e39f6c4a8ca860b49760822bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:53.352797 13402 tablet_bootstrap.cc:492] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9: Bootstrap starting.
I20260429 07:59:53.353082 13403 tablet_bootstrap.cc:492] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: Bootstrap starting.
I20260429 07:59:53.354789 13403 tablet_bootstrap.cc:654] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:53.354877 13402 tablet_bootstrap.cc:654] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:53.355507 13403 log.cc:826] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:53.355687 13402 log.cc:826] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:53.357196 13406 tablet_bootstrap.cc:492] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0: Bootstrap starting.
I20260429 07:59:53.357456 13403 tablet_bootstrap.cc:492] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: No bootstrap required, opened a new log
I20260429 07:59:53.357494 13402 tablet_bootstrap.cc:492] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9: No bootstrap required, opened a new log
I20260429 07:59:53.357647 13403 ts_tablet_manager.cc:1403] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 07:59:53.357661 13402 ts_tablet_manager.cc:1403] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260429 07:59:53.359242 13406 tablet_bootstrap.cc:654] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:53.360105 13406 log.cc:826] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:53.360790 13403 raft_consensus.cc:359] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.361045 13403 raft_consensus.cc:385] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:53.361126 13403 raft_consensus.cc:740] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca352c1be5343f0bd73f7495fb17e11, State: Initialized, Role: FOLLOWER
I20260429 07:59:53.361203 13402 raft_consensus.cc:359] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.361480 13402 raft_consensus.cc:385] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:53.361559 13402 raft_consensus.cc:740] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7e6522ec8647aebcb267b765ef3dc9, State: Initialized, Role: FOLLOWER
I20260429 07:59:53.361606 13403 consensus_queue.cc:260] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.361995 13406 tablet_bootstrap.cc:492] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0: No bootstrap required, opened a new log
I20260429 07:59:53.361955 13402 consensus_queue.cc:260] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.362237 13406 ts_tablet_manager.cc:1403] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260429 07:59:53.362293 13113 heartbeater.cc:499] Master 127.12.12.190:37183 was elected leader, sending a full tablet report...
I20260429 07:59:53.362512 13403 ts_tablet_manager.cc:1434] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 07:59:53.362803 13402 ts_tablet_manager.cc:1434] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 07:59:53.362898 13249 heartbeater.cc:499] Master 127.12.12.190:37183 was elected leader, sending a full tablet report...
W20260429 07:59:53.363149 13114 tablet.cc:2404] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 07:59:53.366106 13406 raft_consensus.cc:359] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.366293 13406 raft_consensus.cc:385] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:53.366391 13406 raft_consensus.cc:740] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4320be2158148dc9db6b266406eace0, State: Initialized, Role: FOLLOWER
I20260429 07:59:53.366789 13406 consensus_queue.cc:260] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.367669 13406 ts_tablet_manager.cc:1434] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 07:59:53.367722 13385 heartbeater.cc:499] Master 127.12.12.190:37183 was elected leader, sending a full tablet report...
I20260429 07:59:53.425125 13410 raft_consensus.cc:493] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 07:59:53.425305 13410 raft_consensus.cc:515] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.426438 13410 leader_election.cc:290] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc7e6522ec8647aebcb267b765ef3dc9 (127.12.12.130:35679), 9ca352c1be5343f0bd73f7495fb17e11 (127.12.12.129:39333)
I20260429 07:59:53.429518 13387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.426638 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:35679 (local address 127.12.12.131:46371)
0429 07:59:53.426769 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.426785 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.426886 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.427248 (+ 362us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.427252 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.427277 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:53.427537 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.427545 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.428470 (+ 925us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.428474 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.429232 (+ 758us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.429240 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.429332 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.429345 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.429382 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.429423 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260429 07:59:53.429867 13411 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.426918 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:39333 (local address 127.12.12.131:40105)
0429 07:59:53.427342 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:53.427354 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:53.427443 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:53.427667 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:53.427670 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:53.427688 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:53.427978 (+ 290us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.427984 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.428795 (+ 811us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.428798 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:53.429481 (+ 683us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:53.429489 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.429600 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.429616 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 07:59:53.429668 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 07:59:53.429714 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260429 07:59:53.430112 13397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.426791 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:46371 (local address 127.12.12.130:35679)
0429 07:59:53.426939 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.426944 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.426959 (+ 15us) server_negotiation.cc:408] Connection header received
0429 07:59:53.427014 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.427020 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.427071 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.427197 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:53.427683 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.428351 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.429355 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.429836 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.429874 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.429927 (+ 53us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.429993 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 07:59:53.430383 13401 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.427079 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:40105 (local address 127.12.12.129:39333)
0429 07:59:53.427248 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.427252 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.427428 (+ 176us) server_negotiation.cc:408] Connection header received
0429 07:59:53.427515 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.427517 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.427565 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.427643 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:53.428100 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.428686 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.429616 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.430114 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.430148 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.430188 (+ 40us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.430262 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 07:59:53.430794 13203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe70b1e39f6c4a8ca860b49760822bcf" candidate_uuid: "b4320be2158148dc9db6b266406eace0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" is_pre_election: true
I20260429 07:59:53.430999 13067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe70b1e39f6c4a8ca860b49760822bcf" candidate_uuid: "b4320be2158148dc9db6b266406eace0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca352c1be5343f0bd73f7495fb17e11" is_pre_election: true
I20260429 07:59:53.431078 13203 raft_consensus.cc:2468] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4320be2158148dc9db6b266406eace0 in term 0.
I20260429 07:59:53.431255 13067 raft_consensus.cc:2468] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4320be2158148dc9db6b266406eace0 in term 0.
I20260429 07:59:53.431540 13273 leader_election.cc:304] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4320be2158148dc9db6b266406eace0, dc7e6522ec8647aebcb267b765ef3dc9; no voters:
I20260429 07:59:53.431811 13410 raft_consensus.cc:2804] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 07:59:53.431890 13410 raft_consensus.cc:493] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 07:59:53.431936 13410 raft_consensus.cc:3060] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:53.433017 13410 raft_consensus.cc:515] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.433311 13410 leader_election.cc:290] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [CANDIDATE]: Term 1 election: Requested vote from peers dc7e6522ec8647aebcb267b765ef3dc9 (127.12.12.130:35679), 9ca352c1be5343f0bd73f7495fb17e11 (127.12.12.129:39333)
I20260429 07:59:53.433641 13203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe70b1e39f6c4a8ca860b49760822bcf" candidate_uuid: "b4320be2158148dc9db6b266406eace0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7e6522ec8647aebcb267b765ef3dc9"
I20260429 07:59:53.433714 13067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe70b1e39f6c4a8ca860b49760822bcf" candidate_uuid: "b4320be2158148dc9db6b266406eace0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca352c1be5343f0bd73f7495fb17e11"
I20260429 07:59:53.433776 13203 raft_consensus.cc:3060] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:53.433856 13067 raft_consensus.cc:3060] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:53.434722 13203 raft_consensus.cc:2468] T fe70b1e39f6c4a8ca860b49760822bcf P dc7e6522ec8647aebcb267b765ef3dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4320be2158148dc9db6b266406eace0 in term 1.
I20260429 07:59:53.434793 13067 raft_consensus.cc:2468] T fe70b1e39f6c4a8ca860b49760822bcf P 9ca352c1be5343f0bd73f7495fb17e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4320be2158148dc9db6b266406eace0 in term 1.
I20260429 07:59:53.435050 13273 leader_election.cc:304] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4320be2158148dc9db6b266406eace0, dc7e6522ec8647aebcb267b765ef3dc9; no voters:
I20260429 07:59:53.435250 13410 raft_consensus.cc:2804] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:53.435461 13410 raft_consensus.cc:697] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [term 1 LEADER]: Becoming Leader. State: Replica: b4320be2158148dc9db6b266406eace0, State: Running, Role: LEADER
I20260429 07:59:53.435757 13410 consensus_queue.cc:237] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } }
I20260429 07:59:53.438876 12919 catalog_manager.cc:5671] T fe70b1e39f6c4a8ca860b49760822bcf P b4320be2158148dc9db6b266406eace0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4320be2158148dc9db6b266406eace0 (127.12.12.131). New cstate: current_term: 1 leader_uuid: "b4320be2158148dc9db6b266406eace0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7e6522ec8647aebcb267b765ef3dc9" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ca352c1be5343f0bd73f7495fb17e11" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4320be2158148dc9db6b266406eace0" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 40961 } health_report { overall_health: HEALTHY } } }
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 07:59:53.465906 13401 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.458048 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.12.12.129:39333)
0429 07:59:53.458205 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.458209 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.458224 (+ 15us) server_negotiation.cc:408] Connection header received
0429 07:59:53.458357 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.458360 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.458414 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.458485 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:53.459245 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.459719 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.460327 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.460457 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.462716 (+ 2259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:53.462740 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:53.462752 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:53.462776 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:53.464402 (+ 1626us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.464819 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.464824 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.464825 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.464869 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.465121 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.465123 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.465125 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.465280 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:53.465463 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.465574 (+ 111us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.465669 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 07:59:53.491254 13401 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.482938 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.12.12.129:39333)
0429 07:59:53.483121 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.483126 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.483304 (+ 178us) server_negotiation.cc:408] Connection header received
0429 07:59:53.483482 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.483488 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.483542 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.483640 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:53.484530 (+ 890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.485088 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.485785 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.485988 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.488391 (+ 2403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:53.488412 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:53.488414 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:53.488441 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:53.489840 (+ 1399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.490268 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.490271 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.490272 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.490320 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.490534 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.490537 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.490538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.490670 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:53.490797 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.491004 (+ 207us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.491123 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[12872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 07:59:53.514818 13401 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:53.507335 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.12.12.129:39333)
0429 07:59:53.507509 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:53.507513 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:53.507529 (+ 16us) server_negotiation.cc:408] Connection header received
0429 07:59:53.507731 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:53.507734 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:53.507781 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:53.507858 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:53.508603 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.509109 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:53.509713 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:53.509833 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:53.512040 (+ 2207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:53.512062 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:53.512065 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:53.512088 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:53.513406 (+ 1318us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.513834 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.513837 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.513838 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.513878 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:53.514120 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:53.514123 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:53.514125 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:53.514258 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:53.514404 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:53.514563 (+ 159us) server_negotiation.cc:300] Negotiation successful
0429 07:59:53.514654 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260429 07:59:53.516319 13027 tablet_service.cc:3018] Rejecting scan request for tablet fe70b1e39f6c4a8ca860b49760822bcf: Uninitialized: clean time has not yet been initialized
I20260429 07:59:53.518183 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 12984
I20260429 07:59:53.524443 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 13120
I20260429 07:59:53.529726 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 13256
I20260429 07:59:53.535609 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 12888
2026-04-29T07:59:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3246 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[13426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[13426](info): set up 2 sockets
Apr 29 07:59:53 dist-test-slave-nhqz krb5kdc[13426](info): commencing operation
krb5kdc: starting...
W20260429 07:59:55.505875 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 07:59:55 dist-test-slave-nhqz krb5kdc[13426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T07:59:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T07:59:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:55.636256 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:46279
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:55.734342 13442 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:55.734616 13442 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:55.734699 13442 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:55.738021 13442 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 07:59:55.738122 13442 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 07:59:55.738152 13442 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:55.738197 13442 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:55.738260 13442 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 07:59:55.738302 13442 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 07:59:55.742568 13442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:46279
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:55.743721 13442 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:55.744580 13442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:55.749954 13450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:55.749958 13447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:55.749959 13448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:55.750284 13442 server_base.cc:1061] running on GCE node
I20260429 07:59:55.750772 13442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:55.751575 13442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:55.752725 13442 hybrid_clock.cc:648] HybridClock initialized: now 1777449595752700 us; error 37 us; skew 500 ppm
Apr 29 07:59:55 dist-test-slave-nhqz krb5kdc[13426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449595, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:55.755239 13442 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 07:59:55.756254 13442 webserver.cc:492] Webserver started at http://127.12.12.190:37853/ using document root <none> and password file <none>
I20260429 07:59:55.756767 13442 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:55.756882 13442 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:55.757112 13442 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:55.758754 13442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "092d7feac12a46e7b485d671d3eb737d"
format_stamp: "Formatted at 2026-04-29 07:59:55 on dist-test-slave-nhqz"
server_key: "e032da6ffd6730ac6d2a5821d270d8df"
server_key_iv: "319fe4383b1c9859c37e6c124923656e"
server_key_version: "encryptionkey@0"
I20260429 07:59:55.759223 13442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "092d7feac12a46e7b485d671d3eb737d"
format_stamp: "Formatted at 2026-04-29 07:59:55 on dist-test-slave-nhqz"
server_key: "e032da6ffd6730ac6d2a5821d270d8df"
server_key_iv: "319fe4383b1c9859c37e6c124923656e"
server_key_version: "encryptionkey@0"
I20260429 07:59:55.762405 13442 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 07:59:55.764459 13457 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:55.765384 13442 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:55.765515 13442 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "092d7feac12a46e7b485d671d3eb737d"
format_stamp: "Formatted at 2026-04-29 07:59:55 on dist-test-slave-nhqz"
server_key: "e032da6ffd6730ac6d2a5821d270d8df"
server_key_iv: "319fe4383b1c9859c37e6c124923656e"
server_key_version: "encryptionkey@0"
I20260429 07:59:55.765609 13442 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:55.775388 13442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:55.778103 13442 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:55.778314 13442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:55.785828 13442 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:46279
I20260429 07:59:55.786106 13519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:46279 every 8 connection(s)
I20260429 07:59:55.786857 13442 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 07:59:55.789516 13520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:55.791348 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 13442
I20260429 07:59:55.791478 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 07:59:55.791675 12338 external_mini_cluster.cc:1468] Setting key ca18f045d74d1a864700720bf85af2f5
I20260429 07:59:55.794617 13520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: Bootstrap starting.
I20260429 07:59:55.796746 13520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:55.797585 13520 log.cc:826] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: Log is configured to *not* fsync() on all Append() calls
Apr 29 07:59:55 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:55.799499 13520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: No bootstrap required, opened a new log
I20260429 07:59:55.803372 13520 raft_consensus.cc:359] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092d7feac12a46e7b485d671d3eb737d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46279 } }
I20260429 07:59:55.803576 13520 raft_consensus.cc:385] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:55.803635 13520 raft_consensus.cc:740] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092d7feac12a46e7b485d671d3eb737d, State: Initialized, Role: FOLLOWER
I20260429 07:59:55.804003 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:55.792857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.12.12.190:46279)
0429 07:59:55.793230 (+ 373us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:55.793240 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:55.793275 (+ 35us) server_negotiation.cc:408] Connection header received
0429 07:59:55.793858 (+ 583us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:55.793880 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:55.794182 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:55.794487 (+ 305us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:55.795292 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:55.796329 (+ 1037us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:55.797073 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:55.797327 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:55.799534 (+ 2207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:55.799556 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:55.799568 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:55.799598 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:55.801572 (+ 1974us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:55.801985 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:55.801992 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:55.802000 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:55.802103 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:55.802307 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:55.802311 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:55.802313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:55.802682 (+ 369us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:55.802860 (+ 178us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:55.803081 (+ 221us) server_negotiation.cc:300] Negotiation successful
0429 07:59:55.803298 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260429 07:59:55.804047 13520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092d7feac12a46e7b485d671d3eb737d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46279 } }
I20260429 07:59:55.804209 13520 raft_consensus.cc:399] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 07:59:55.804266 13520 raft_consensus.cc:493] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 07:59:55.804347 13520 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:55.805294 13520 raft_consensus.cc:515] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092d7feac12a46e7b485d671d3eb737d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46279 } }
I20260429 07:59:55.805606 13520 leader_election.cc:304] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092d7feac12a46e7b485d671d3eb737d; no voters:
I20260429 07:59:55.805912 13520 leader_election.cc:290] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 07:59:55.806030 13525 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:55.806273 13525 raft_consensus.cc:697] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [term 1 LEADER]: Becoming Leader. State: Replica: 092d7feac12a46e7b485d671d3eb737d, State: Running, Role: LEADER
I20260429 07:59:55.806618 13525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092d7feac12a46e7b485d671d3eb737d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46279 } }
I20260429 07:59:55.807032 13520 sys_catalog.cc:565] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [sys.catalog]: configured and running, proceeding with master startup.
I20260429 07:59:55.808241 13527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092d7feac12a46e7b485d671d3eb737d. Latest consensus state: current_term: 1 leader_uuid: "092d7feac12a46e7b485d671d3eb737d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092d7feac12a46e7b485d671d3eb737d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46279 } } }
I20260429 07:59:55.808245 13526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092d7feac12a46e7b485d671d3eb737d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092d7feac12a46e7b485d671d3eb737d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46279 } } }
I20260429 07:59:55.808471 13526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:55.808403 13527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:55.808861 13534 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 07:59:55.811668 13534 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 07:59:55.817253 13534 catalog_manager.cc:1357] Generated new cluster ID: ce8e2fda1bc34d8a831a102c2f5ed1ca
I20260429 07:59:55.817332 13534 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 07:59:55.824273 13534 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 07:59:55.825235 13534 catalog_manager.cc:1514] Loading token signing keys...
I20260429 07:59:55.842470 13534 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: Generated new TSK 0
I20260429 07:59:55.843233 13534 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:55.890071 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:46279
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:55.989935 13548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:55.990144 13548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:55.990239 13548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:55.993454 13548 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:55.993508 13548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:55.993599 13548 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 07:59:55.993693 13548 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 07:59:55.997742 13548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:55.998746 13548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:55.999574 13548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:56.005800 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:56.005772 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:56.005853 13548 server_base.cc:1061] running on GCE node
W20260429 07:59:56.005781 13556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:56.006423 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:56.006943 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:56.008162 13548 hybrid_clock.cc:648] HybridClock initialized: now 1777449596008154 us; error 26 us; skew 500 ppm
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:56.010756 13548 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 07:59:56.011729 13548 webserver.cc:492] Webserver started at http://127.12.12.129:38161/ using document root <none> and password file <none>
I20260429 07:59:56.012253 13548 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:56.012326 13548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:56.012573 13548 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:56.014305 13548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "0bf99824798a4af795e330a314248eb1"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "0542ee274cdd46e2c0672a04c6129aaa"
server_key_iv: "989f53b1ebb99b4aa1e7dea943a2d9a9"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.014768 13548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "0bf99824798a4af795e330a314248eb1"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "0542ee274cdd46e2c0672a04c6129aaa"
server_key_iv: "989f53b1ebb99b4aa1e7dea943a2d9a9"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.017959 13548 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 07:59:56.019927 13563 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:56.020895 13548 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 07:59:56.021011 13548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "0bf99824798a4af795e330a314248eb1"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "0542ee274cdd46e2c0672a04c6129aaa"
server_key_iv: "989f53b1ebb99b4aa1e7dea943a2d9a9"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.021109 13548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:56.039351 13548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:56.041751 13548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:56.041944 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:56.043437 13548 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:56.043536 13548 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:56.043630 13548 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:56.043664 13548 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:56.055023 13548 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:38267
I20260429 07:59:56.055079 13678 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:38267 every 8 connection(s)
I20260429 07:59:56.055413 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.043902 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:48957 (local address 127.12.12.190:46279)
0429 07:59:56.044056 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.044061 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.045178 (+ 1117us) server_negotiation.cc:408] Connection header received
0429 07:59:56.046022 (+ 844us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.046029 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.046093 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.046156 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.047440 (+ 1284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.047977 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.048697 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.048908 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.051514 (+ 2606us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.051534 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.051537 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.051564 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.053261 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.053887 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.053892 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.053895 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.053947 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.054245 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.054247 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.054248 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.054443 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.054520 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.055106 (+ 586us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.055253 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 07:59:56.055946 13572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.044207 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.12.12.129:48957)
0429 07:59:56.045024 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.045092 (+ 68us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.045793 (+ 701us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.046458 (+ 665us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.046469 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.046803 (+ 334us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.047251 (+ 448us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.047261 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.048111 (+ 850us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.048114 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.048566 (+ 452us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.048574 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.048780 (+ 206us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.049376 (+ 596us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.049414 (+ 38us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.051366 (+ 1952us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.053395 (+ 2029us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.053402 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.053413 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.053766 (+ 353us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.054044 (+ 278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.054048 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.054050 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.054153 (+ 103us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.054555 (+ 402us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.054564 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.054812 (+ 248us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.055030 (+ 218us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.055249 (+ 219us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":124,"threads_started":1}
I20260429 07:59:56.056093 13548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 07:59:56.057636 13679 heartbeater.cc:344] Connected to a master server at 127.12.12.190:46279
I20260429 07:59:56.057796 13679 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:56.058296 13679 heartbeater.cc:507] Master 127.12.12.190:46279 requested a full tablet report, sending...
I20260429 07:59:56.059734 13474 ts_manager.cc:194] Registered new tserver with Master: 0bf99824798a4af795e330a314248eb1 (127.12.12.129:38267)
I20260429 07:59:56.061149 13474 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:48957
I20260429 07:59:56.065363 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.059250 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:42019 (local address 127.12.12.190:46279)
0429 07:59:56.059375 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.059379 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.059597 (+ 218us) server_negotiation.cc:408] Connection header received
0429 07:59:56.059671 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.059675 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.059718 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.059800 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.060318 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.061198 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.061882 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.062086 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.062787 (+ 701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.062801 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.062805 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.062833 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.063984 (+ 1151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.064373 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.064376 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.064391 (+ 15us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.064429 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.064659 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.064663 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.064665 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.064784 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.064917 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.065148 (+ 231us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.065238 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 07:59:56.065384 13685 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.059165 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.12.12.129:42019)
0429 07:59:56.059513 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.059524 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.059601 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.059842 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.059848 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.060005 (+ 157us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.060206 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.060211 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.061331 (+ 1120us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.061335 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.061773 (+ 438us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.061781 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.061898 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.062381 (+ 483us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.062394 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.062672 (+ 278us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.064097 (+ 1425us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.064101 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.064103 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.064277 (+ 174us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.064531 (+ 254us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.064533 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.064535 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.064575 (+ 40us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.064908 (+ 333us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.064911 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.064988 (+ 77us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.065088 (+ 100us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.065211 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":91,"threads_started":1}
I20260429 07:59:56.065559 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 13548
I20260429 07:59:56.065663 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 07:59:56.065889 12338 external_mini_cluster.cc:1468] Setting key 2f68c40d66f76cc8ea4d002eec38b080
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:56.107599 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:46279
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:56.206411 13690 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:56.206655 13690 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:56.206748 13690 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:56.210251 13690 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:56.210345 13690 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:56.210428 13690 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 07:59:56.210528 13690 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 07:59:56.214819 13690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:56.215919 13690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:56.216775 13690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:56.222991 13695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:56.223024 13696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:56.223084 13698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:56.223165 13690 server_base.cc:1061] running on GCE node
I20260429 07:59:56.223531 13690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:56.224066 13690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:56.225207 13690 hybrid_clock.cc:648] HybridClock initialized: now 1777449596225195 us; error 24 us; skew 500 ppm
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:56.227762 13690 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 07:59:56.228770 13690 webserver.cc:492] Webserver started at http://127.12.12.130:36823/ using document root <none> and password file <none>
I20260429 07:59:56.229389 13690 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:56.229439 13690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:56.229640 13690 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:56.231307 13690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "55e574dc4c8a42fb98bb2333bb84fdf0"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "964fb39526544732c0fe34dc07acb603"
server_key_iv: "91e5529a94364ed7cda5be6596b0fe20"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.231781 13690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "55e574dc4c8a42fb98bb2333bb84fdf0"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "964fb39526544732c0fe34dc07acb603"
server_key_iv: "91e5529a94364ed7cda5be6596b0fe20"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.235155 13690 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260429 07:59:56.237221 13705 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:56.238113 13690 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:56.238246 13690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "55e574dc4c8a42fb98bb2333bb84fdf0"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "964fb39526544732c0fe34dc07acb603"
server_key_iv: "91e5529a94364ed7cda5be6596b0fe20"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.238340 13690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:56.257375 13690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:56.259788 13690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:56.259977 13690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:56.261446 13690 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:56.261513 13690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:56.261579 13690 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:56.261617 13690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:56.272933 13690 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:46009
I20260429 07:59:56.272981 13820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:46009 every 8 connection(s)
I20260429 07:59:56.273851 13714 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.262175 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.12.12.130:42689)
0429 07:59:56.262669 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.262716 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.263716 (+ 1000us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.264441 (+ 725us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.264453 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.265009 (+ 556us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.265699 (+ 690us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.265716 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.266652 (+ 936us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.266656 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.267045 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.267052 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.267269 (+ 217us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.267799 (+ 530us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.267815 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.269555 (+ 1740us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.271654 (+ 2099us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.271662 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.271673 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.271931 (+ 258us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.272162 (+ 231us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.272164 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.272165 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.272236 (+ 71us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.272622 (+ 386us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.272628 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.272884 (+ 256us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.273089 (+ 205us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.273268 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260429 07:59:56.273869 13690 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 07:59:56.274533 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.262925 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:42689 (local address 127.12.12.190:46279)
0429 07:59:56.263100 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.263104 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.263120 (+ 16us) server_negotiation.cc:408] Connection header received
0429 07:59:56.263919 (+ 799us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.263922 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.263966 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.264233 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.265873 (+ 1640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.266528 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.267196 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.267395 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.269707 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.269733 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.269736 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.269768 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.271516 (+ 1748us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.272020 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.272024 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.272025 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.272075 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.272318 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.272321 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.272323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.272498 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.272628 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.274244 (+ 1616us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.274374 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260429 07:59:56.275378 13821 heartbeater.cc:344] Connected to a master server at 127.12.12.190:46279
I20260429 07:59:56.275537 13821 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:56.275970 13821 heartbeater.cc:507] Master 127.12.12.190:46279 requested a full tablet report, sending...
I20260429 07:59:56.276973 13473 ts_manager.cc:194] Registered new tserver with Master: 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009)
I20260429 07:59:56.277673 13473 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:42689
I20260429 07:59:56.282810 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 13690
I20260429 07:59:56.282966 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 07:59:56.283231 13827 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.277575 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.12.12.130:48151)
0429 07:59:56.277833 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.277844 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.277947 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.278191 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.278194 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.278361 (+ 167us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.278549 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.278554 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.279203 (+ 649us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.279208 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.279580 (+ 372us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.279586 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.279709 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.280239 (+ 530us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.280250 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.280503 (+ 253us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.281961 (+ 1458us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.281965 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.281967 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.282127 (+ 160us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.282384 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.282386 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.282388 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.282427 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.282775 (+ 348us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.282778 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.282856 (+ 78us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.283002 (+ 146us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.283102 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260429 07:59:56.283262 12338 external_mini_cluster.cc:1468] Setting key bc6599bf0c7e6d18ead41ef62d869c29
I20260429 07:59:56.283347 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.277642 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:48151 (local address 127.12.12.190:46279)
0429 07:59:56.277756 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.277760 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.277908 (+ 148us) server_negotiation.cc:408] Connection header received
0429 07:59:56.278046 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.278049 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.278093 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.278177 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.278659 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.279101 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.279692 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.279890 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.280622 (+ 732us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.280638 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.280642 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.280671 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.281852 (+ 1181us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.282232 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.282237 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.282239 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.282284 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.282512 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.282516 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.282518 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.282683 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.282785 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.283075 (+ 290us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.283185 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:56.327725 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:46279
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:56.426620 13832 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:56.426798 13832 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:56.426864 13832 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:56.430121 13832 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:56.430186 13832 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:56.430235 13832 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 07:59:56.430281 13832 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 07:59:56.434258 13832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:56.435119 13832 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:56.435827 13832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:56.441771 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:56.441926 13832 server_base.cc:1061] running on GCE node
W20260429 07:59:56.441777 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:56.441862 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:56.442390 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:56.442963 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:56.444111 13832 hybrid_clock.cc:648] HybridClock initialized: now 1777449596444098 us; error 27 us; skew 500 ppm
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:56.446626 13832 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 07:59:56.447615 13832 webserver.cc:492] Webserver started at http://127.12.12.131:33139/ using document root <none> and password file <none>
I20260429 07:59:56.448092 13832 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:56.448137 13832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:56.448279 13832 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:56.449899 13832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "da8f856658b54548a7f1776107b8ffb5"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "e6fa6fbb5eeb24da243ff0005f2ae830"
server_key_iv: "62dc3edc895d1f5e95fdf07158ca1bd0"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.450304 13832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "da8f856658b54548a7f1776107b8ffb5"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "e6fa6fbb5eeb24da243ff0005f2ae830"
server_key_iv: "62dc3edc895d1f5e95fdf07158ca1bd0"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.453222 13832 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 07:59:56.455168 13847 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:56.456008 13832 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:56.456127 13832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "da8f856658b54548a7f1776107b8ffb5"
format_stamp: "Formatted at 2026-04-29 07:59:56 on dist-test-slave-nhqz"
server_key: "e6fa6fbb5eeb24da243ff0005f2ae830"
server_key_iv: "62dc3edc895d1f5e95fdf07158ca1bd0"
server_key_version: "encryptionkey@0"
I20260429 07:59:56.456243 13832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:56.474543 13832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:56.476998 13832 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:56.477195 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:56.478606 13832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:56.478677 13832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:56.478744 13832 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:56.478813 13832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:56.489890 13832 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:33193
I20260429 07:59:56.489949 13962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:33193 every 8 connection(s)
I20260429 07:59:56.490496 13856 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.479398 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.12.12.131:56539)
0429 07:59:56.479791 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.479827 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.480573 (+ 746us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.481124 (+ 551us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.481134 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.481519 (+ 385us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.482121 (+ 602us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.482135 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.482952 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.482969 (+ 17us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.483538 (+ 569us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.483545 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.483735 (+ 190us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.484291 (+ 556us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.484315 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.486302 (+ 1987us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.488130 (+ 1828us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.488136 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.488147 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.488431 (+ 284us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.488696 (+ 265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.488699 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.488701 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.488871 (+ 170us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.489231 (+ 360us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.489239 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.489433 (+ 194us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.489657 (+ 224us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.489859 (+ 202us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260429 07:59:56.490864 13832 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 07:59:56.491125 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.479180 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:56539 (local address 127.12.12.190:46279)
0429 07:59:56.479340 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.479346 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.479935 (+ 589us) server_negotiation.cc:408] Connection header received
0429 07:59:56.480758 (+ 823us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.480762 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.480858 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.480961 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.482268 (+ 1307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.482823 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.483652 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.483836 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.486447 (+ 2611us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.486464 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.486466 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.486489 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.488015 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.488537 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.488541 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.488544 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.488593 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.488970 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.488974 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.488976 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.489114 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.489222 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.490874 (+ 1652us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.491014 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 07:59:56.492560 13963 heartbeater.cc:344] Connected to a master server at 127.12.12.190:46279
I20260429 07:59:56.492738 13963 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:56.493033 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 13832
I20260429 07:59:56.493132 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 07:59:56.493359 13963 heartbeater.cc:507] Master 127.12.12.190:46279 requested a full tablet report, sending...
I20260429 07:59:56.493391 12338 external_mini_cluster.cc:1468] Setting key ccd0459174c10ef00e15da2a7500c21a
I20260429 07:59:56.494398 13474 ts_manager.cc:194] Registered new tserver with Master: da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193)
I20260429 07:59:56.495083 13474 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:56539
I20260429 07:59:56.495359 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 07:59:56.501106 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.494114 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:58599 (local address 127.12.12.190:46279)
0429 07:59:56.494237 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.494240 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.494521 (+ 281us) server_negotiation.cc:408] Connection header received
0429 07:59:56.494688 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.494692 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.494739 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.494830 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.495396 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.496070 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.496856 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.497329 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.497918 (+ 589us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.497936 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.497942 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.497971 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.499685 (+ 1714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.500061 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.500066 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.500071 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.500124 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.500395 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.500397 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.500399 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.500516 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.500590 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.500923 (+ 333us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.501002 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 07:59:56.501258 13969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.494119 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.12.12.131:58599)
0429 07:59:56.494416 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.494437 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.494589 (+ 152us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.494843 (+ 254us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.494847 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.495018 (+ 171us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.495255 (+ 237us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.495262 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.496192 (+ 930us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.496195 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.496707 (+ 512us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.496715 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.496868 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.497365 (+ 497us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.497392 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.497774 (+ 382us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.499795 (+ 2021us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.499798 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.499799 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.499958 (+ 159us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.500227 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.500231 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.500233 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.500293 (+ 60us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.500615 (+ 322us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.500621 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.500714 (+ 93us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.500858 (+ 144us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.500992 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:56.518960 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.512469 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.12.12.190:46279)
0429 07:59:56.512600 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.512604 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.512615 (+ 11us) server_negotiation.cc:408] Connection header received
0429 07:59:56.512648 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.512651 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.512690 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.512758 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.513551 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.514000 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.514553 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.514697 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.516598 (+ 1901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.516613 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.516615 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.516636 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.517776 (+ 1140us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.518157 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.518159 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.518161 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.518198 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.518384 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.518386 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.518387 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.518504 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.518602 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.518767 (+ 165us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.518845 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 07:59:56.521499 13474 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 07:59:56.523783 13474 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 07:59:56.536019 13980 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.531790 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:33193 (local address 127.0.0.1:35358)
0429 07:59:56.532116 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.532175 (+ 59us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.532322 (+ 147us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.532956 (+ 634us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.532962 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.532987 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.533275 (+ 288us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.533282 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.534623 (+ 1341us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.534627 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.535559 (+ 932us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.535568 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.535688 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.535737 (+ 49us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.535817 (+ 80us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.535884 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260429 07:59:56.536963 13983 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.532780 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:38267 (local address 127.0.0.1:60890)
0429 07:59:56.533407 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.533419 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.533494 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.534108 (+ 614us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.534112 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.534129 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.534387 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.534393 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.535730 (+ 1337us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.535734 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.536569 (+ 835us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.536579 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.536677 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.536689 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.536734 (+ 45us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.536788 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":432,"threads_started":1}
I20260429 07:59:56.536865 13982 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.532469 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46009 (local address 127.0.0.1:49326)
0429 07:59:56.532686 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.532701 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.532777 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.534038 (+ 1261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.534042 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.534060 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.534296 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.534303 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.535698 (+ 1395us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.535701 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.536462 (+ 761us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.536470 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.536597 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.536611 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.536660 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.536710 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260429 07:59:56.537236 13981 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.531939 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.12.12.131:33193)
0429 07:59:56.532494 (+ 555us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.532500 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.532517 (+ 17us) server_negotiation.cc:408] Connection header received
0429 07:59:56.532612 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.532618 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.532795 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.532961 (+ 166us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.533424 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.534505 (+ 1081us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.535785 (+ 1280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.536625 (+ 840us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.536742 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.536800 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.536904 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":104,"threads_started":1}
I20260429 07:59:56.537777 13985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.533236 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.12.12.130:46009)
0429 07:59:56.533710 (+ 474us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.533716 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.533741 (+ 25us) server_negotiation.cc:408] Connection header received
0429 07:59:56.533801 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.533804 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.533917 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.534052 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.534437 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.535560 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.536598 (+ 1038us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.537455 (+ 857us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.537570 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.537619 (+ 49us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.537671 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":98,"threads_started":1}
I20260429 07:59:56.537868 13984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.533236 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.12.12.129:38267)
0429 07:59:56.533715 (+ 479us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.533721 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.533739 (+ 18us) server_negotiation.cc:408] Connection header received
0429 07:59:56.533811 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.533816 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.533968 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.534518 (+ 550us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.534689 (+ 171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.535569 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.536934 (+ 1365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.537511 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.537619 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.537672 (+ 53us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.537741 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":86,"threads_started":1}
I20260429 07:59:56.539852 13897 tablet_service.cc:1511] Processing CreateTablet for tablet 89856be9d8844f76a4975a4d31a5166e (DEFAULT_TABLE table=test-table [id=1169621ff760433c89cab2e4f82edec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:56.540220 13613 tablet_service.cc:1511] Processing CreateTablet for tablet 89856be9d8844f76a4975a4d31a5166e (DEFAULT_TABLE table=test-table [id=1169621ff760433c89cab2e4f82edec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:56.540220 13755 tablet_service.cc:1511] Processing CreateTablet for tablet 89856be9d8844f76a4975a4d31a5166e (DEFAULT_TABLE table=test-table [id=1169621ff760433c89cab2e4f82edec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:56.540753 13897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89856be9d8844f76a4975a4d31a5166e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.541083 13755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89856be9d8844f76a4975a4d31a5166e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.541085 13613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89856be9d8844f76a4975a4d31a5166e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.546455 13986 tablet_bootstrap.cc:492] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Bootstrap starting.
I20260429 07:59:56.547344 13988 tablet_bootstrap.cc:492] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Bootstrap starting.
I20260429 07:59:56.547345 13987 tablet_bootstrap.cc:492] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Bootstrap starting.
I20260429 07:59:56.548977 13986 tablet_bootstrap.cc:654] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.549311 13988 tablet_bootstrap.cc:654] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.549312 13987 tablet_bootstrap.cc:654] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.550100 13986 log.cc:826] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:56.550489 13988 log.cc:826] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:56.550489 13987 log.cc:826] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:56.551832 13986 tablet_bootstrap.cc:492] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: No bootstrap required, opened a new log
I20260429 07:59:56.552033 13986 ts_tablet_manager.cc:1403] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260429 07:59:56.552143 13988 tablet_bootstrap.cc:492] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: No bootstrap required, opened a new log
I20260429 07:59:56.552397 13987 tablet_bootstrap.cc:492] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: No bootstrap required, opened a new log
I20260429 07:59:56.552416 13988 ts_tablet_manager.cc:1403] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260429 07:59:56.552601 13987 ts_tablet_manager.cc:1403] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260429 07:59:56.554503 13986 raft_consensus.cc:359] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.554692 13986 raft_consensus.cc:385] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.554762 13986 raft_consensus.cc:740] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8f856658b54548a7f1776107b8ffb5, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.555168 13986 consensus_queue.cc:260] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.555423 13988 raft_consensus.cc:359] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.555424 13987 raft_consensus.cc:359] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.555603 13987 raft_consensus.cc:385] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.555680 13988 raft_consensus.cc:385] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.555687 13987 raft_consensus.cc:740] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf99824798a4af795e330a314248eb1, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.555749 13988 raft_consensus.cc:740] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e574dc4c8a42fb98bb2333bb84fdf0, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.555768 13963 heartbeater.cc:499] Master 127.12.12.190:46279 was elected leader, sending a full tablet report...
I20260429 07:59:56.556070 13987 consensus_queue.cc:260] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.556070 13988 consensus_queue.cc:260] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.556895 13988 ts_tablet_manager.cc:1434] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260429 07:59:56.556895 13987 ts_tablet_manager.cc:1434] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260429 07:59:56.557112 13986 ts_tablet_manager.cc:1434] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260429 07:59:56.557199 13679 heartbeater.cc:499] Master 127.12.12.190:46279 was elected leader, sending a full tablet report...
I20260429 07:59:56.558104 13821 heartbeater.cc:499] Master 127.12.12.190:46279 was elected leader, sending a full tablet report...
W20260429 07:59:56.558153 13680 tablet.cc:2404] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 07:59:56.582245 13994 raft_consensus.cc:493] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 07:59:56.582407 13994 raft_consensus.cc:515] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.583410 13994 leader_election.cc:290] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193), 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009)
I20260429 07:59:56.587338 13996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.584069 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46009 (local address 127.12.12.129:54129)
0429 07:59:56.584304 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.584317 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.584400 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.584642 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.584645 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.584658 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.584949 (+ 291us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.584955 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.585900 (+ 945us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.585903 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.586857 (+ 954us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.586863 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.586999 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.587017 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.587075 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.587132 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260429 07:59:56.587338 13995 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.583692 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:33193 (local address 127.12.12.129:38741)
0429 07:59:56.583947 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.583961 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.584080 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.584513 (+ 433us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.584516 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.584532 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.584860 (+ 328us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.584867 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.585900 (+ 1033us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.585903 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.586849 (+ 946us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.586858 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.586999 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.587017 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.587075 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.587139 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260429 07:59:56.587733 13985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.584211 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:54129 (local address 127.12.12.130:46009)
0429 07:59:56.584339 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.584342 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.584357 (+ 15us) server_negotiation.cc:408] Connection header received
0429 07:59:56.584508 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.584511 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.584549 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.584612 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.585062 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.585786 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.586996 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.587487 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.587515 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.587575 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.587621 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 07:59:56.587733 13981 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.583823 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:38741 (local address 127.12.12.131:33193)
0429 07:59:56.584009 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.584013 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.584030 (+ 17us) server_negotiation.cc:408] Connection header received
0429 07:59:56.584254 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.584258 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.584314 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.584370 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.584996 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.585786 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.586996 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.587482 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.587515 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.587575 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.587621 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 07:59:56.588277 13917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89856be9d8844f76a4975a4d31a5166e" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8f856658b54548a7f1776107b8ffb5" is_pre_election: true
I20260429 07:59:56.588285 13760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89856be9d8844f76a4975a4d31a5166e" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" is_pre_election: true
I20260429 07:59:56.588538 13760 raft_consensus.cc:2468] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 0.
I20260429 07:59:56.588547 13917 raft_consensus.cc:2468] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 0.
I20260429 07:59:56.588989 13564 leader_election.cc:304] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf99824798a4af795e330a314248eb1, da8f856658b54548a7f1776107b8ffb5; no voters:
I20260429 07:59:56.589279 13994 raft_consensus.cc:2804] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 07:59:56.589344 13994 raft_consensus.cc:493] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 07:59:56.589376 13994 raft_consensus.cc:3060] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.590420 13994 raft_consensus.cc:515] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.590821 13994 leader_election.cc:290] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 election: Requested vote from peers da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193), 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009)
I20260429 07:59:56.591166 13917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89856be9d8844f76a4975a4d31a5166e" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8f856658b54548a7f1776107b8ffb5"
I20260429 07:59:56.591229 13760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89856be9d8844f76a4975a4d31a5166e" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0"
I20260429 07:59:56.591305 13917 raft_consensus.cc:3060] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.591348 13760 raft_consensus.cc:3060] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.592234 13917 raft_consensus.cc:2468] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 1.
I20260429 07:59:56.592309 13760 raft_consensus.cc:2468] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 1.
I20260429 07:59:56.592577 13564 leader_election.cc:304] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf99824798a4af795e330a314248eb1, da8f856658b54548a7f1776107b8ffb5; no voters:
I20260429 07:59:56.592890 13994 raft_consensus.cc:2804] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:56.593091 13994 raft_consensus.cc:697] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf99824798a4af795e330a314248eb1, State: Running, Role: LEADER
I20260429 07:59:56.593387 13994 consensus_queue.cc:237] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } }
I20260429 07:59:56.596320 13472 catalog_manager.cc:5671] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bf99824798a4af795e330a314248eb1 (127.12.12.129). New cstate: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } health_report { overall_health: HEALTHY } } }
I20260429 07:59:56.616035 13984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.613398 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.12.12.129:38267)
0429 07:59:56.613528 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.613531 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.613543 (+ 12us) server_negotiation.cc:408] Connection header received
0429 07:59:56.613680 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.613682 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.613731 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.613836 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 07:59:56.614182 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.614648 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.615240 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.615364 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.615403 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.615705 (+ 302us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.615779 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.615874 (+ 95us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.615912 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 07:59:56.621081 13760 raft_consensus.cc:1275] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Refusing update from remote peer 0bf99824798a4af795e330a314248eb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 07:59:56.621081 13917 raft_consensus.cc:1275] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Refusing update from remote peer 0bf99824798a4af795e330a314248eb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 07:59:56.621749 13994 consensus_queue.cc:1048] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 07:59:56.621939 13997 consensus_queue.cc:1048] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 07:59:56.630146 14001 mvcc.cc:204] Tried to move back new op lower bound from 7280433547753000960 to 7280433547649179648. Current Snapshot: MvccSnapshot[applied={T|T < 7280433547753000960}]
I20260429 07:59:56.631392 14002 mvcc.cc:204] Tried to move back new op lower bound from 7280433547753000960 to 7280433547649179648. Current Snapshot: MvccSnapshot[applied={T|T < 7280433547753000960}]
I20260429 07:59:56.637794 13473 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 07:59:56.637982 13473 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 07:59:56.640357 13473 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: Sending DeleteTablet for 3 replicas of tablet 89856be9d8844f76a4975a4d31a5166e
I20260429 07:59:56.641134 13897 tablet_service.cc:1558] Processing DeleteTablet for tablet 89856be9d8844f76a4975a4d31a5166e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 07:59:56 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:35358
I20260429 07:59:56.641233 13755 tablet_service.cc:1558] Processing DeleteTablet for tablet 89856be9d8844f76a4975a4d31a5166e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 07:59:56 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49326
I20260429 07:59:56.641788 14011 tablet_replica.cc:333] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: stopping tablet replica
I20260429 07:59:56.641755 13473 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 07:59:56.641810 13613 tablet_service.cc:1558] Processing DeleteTablet for tablet 89856be9d8844f76a4975a4d31a5166e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 07:59:56 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:60890
I20260429 07:59:56.642091 14011 raft_consensus.cc:2243] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260429 07:59:56.642146 13473 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 07:59:56.642241 14010 tablet_replica.cc:333] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: stopping tablet replica
I20260429 07:59:56.642364 14011 raft_consensus.cc:2272] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 07:59:56.642457 14010 raft_consensus.cc:2243] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 07:59:56.642716 14010 raft_consensus.cc:2272] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 07:59:56.643821 14011 ts_tablet_manager.cc:1916] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 07:59:56.644291 14010 ts_tablet_manager.cc:1916] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 07:59:56.646672 14011 ts_tablet_manager.cc:1929] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 07:59:56.646762 14010 ts_tablet_manager.cc:1929] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 07:59:56.646847 14011 log.cc:1199] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/89856be9d8844f76a4975a4d31a5166e
I20260429 07:59:56.646908 14010 log.cc:1199] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/89856be9d8844f76a4975a4d31a5166e
I20260429 07:59:56.647212 14011 ts_tablet_manager.cc:1950] T 89856be9d8844f76a4975a4d31a5166e P 55e574dc4c8a42fb98bb2333bb84fdf0: Deleting consensus metadata
I20260429 07:59:56.647296 14010 ts_tablet_manager.cc:1950] T 89856be9d8844f76a4975a4d31a5166e P da8f856658b54548a7f1776107b8ffb5: Deleting consensus metadata
I20260429 07:59:56.648331 13458 catalog_manager.cc:5002] TS da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193): tablet 89856be9d8844f76a4975a4d31a5166e (table test-table [id=1169621ff760433c89cab2e4f82edec7]) successfully deleted
I20260429 07:59:56.648329 13613 tablet_service.cc:1511] Processing CreateTablet for tablet e5a825d2566049369318c1c5f9b8958e (DEFAULT_TABLE table=test-table [id=0e4b0c7c4ff742f889595173e50a2148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:56.648586 13613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a825d2566049369318c1c5f9b8958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.648613 13897 tablet_service.cc:1511] Processing CreateTablet for tablet e5a825d2566049369318c1c5f9b8958e (DEFAULT_TABLE table=test-table [id=0e4b0c7c4ff742f889595173e50a2148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:56.648864 13897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a825d2566049369318c1c5f9b8958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.650261 13987 tablet_bootstrap.cc:492] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Bootstrap starting.
I20260429 07:59:56.650942 13986 tablet_bootstrap.cc:492] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Bootstrap starting.
I20260429 07:59:56.651299 13987 tablet_bootstrap.cc:654] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.651358 13755 tablet_service.cc:1511] Processing CreateTablet for tablet e5a825d2566049369318c1c5f9b8958e (DEFAULT_TABLE table=test-table [id=0e4b0c7c4ff742f889595173e50a2148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 07:59:56.651587 13755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a825d2566049369318c1c5f9b8958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.652101 13986 tablet_bootstrap.cc:654] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.652668 13987 tablet_bootstrap.cc:492] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: No bootstrap required, opened a new log
I20260429 07:59:56.652750 13987 ts_tablet_manager.cc:1403] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 07:59:56.653231 13986 tablet_bootstrap.cc:492] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: No bootstrap required, opened a new log
I20260429 07:59:56.653218 13987 raft_consensus.cc:359] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.653291 13986 ts_tablet_manager.cc:1403] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:56.653343 13987 raft_consensus.cc:385] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.653374 13987 raft_consensus.cc:740] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf99824798a4af795e330a314248eb1, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.653523 13987 consensus_queue.cc:260] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.653609 13458 catalog_manager.cc:5002] TS 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009): tablet 89856be9d8844f76a4975a4d31a5166e (table test-table [id=1169621ff760433c89cab2e4f82edec7]) successfully deleted
I20260429 07:59:56.653759 13987 ts_tablet_manager.cc:1434] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 07:59:56.653749 13986 raft_consensus.cc:359] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.653836 13986 raft_consensus.cc:385] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.653869 13986 raft_consensus.cc:740] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8f856658b54548a7f1776107b8ffb5, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.653985 13986 consensus_queue.cc:260] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.654263 13986 ts_tablet_manager.cc:1434] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 07:59:56.654292 14012 tablet_replica.cc:333] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: stopping tablet replica
I20260429 07:59:56.654448 13988 tablet_bootstrap.cc:492] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Bootstrap starting.
I20260429 07:59:56.654524 14012 raft_consensus.cc:2243] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 1 LEADER]: Raft consensus shutting down.
I20260429 07:59:56.654866 14012 raft_consensus.cc:2272] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 07:59:56.655565 13988 tablet_bootstrap.cc:654] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.656091 14012 ts_tablet_manager.cc:1916] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 07:59:56.657797 13988 tablet_bootstrap.cc:492] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: No bootstrap required, opened a new log
I20260429 07:59:56.657907 13988 ts_tablet_manager.cc:1403] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 07:59:56.658488 14012 ts_tablet_manager.cc:1929] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 07:59:56.658524 13988 raft_consensus.cc:359] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.658638 13988 raft_consensus.cc:385] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.658674 14012 log.cc:1199] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/89856be9d8844f76a4975a4d31a5166e
I20260429 07:59:56.658684 13988 raft_consensus.cc:740] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e574dc4c8a42fb98bb2333bb84fdf0, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.658811 13988 consensus_queue.cc:260] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.659091 13988 ts_tablet_manager.cc:1434] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 07:59:56.659137 14012 ts_tablet_manager.cc:1950] T 89856be9d8844f76a4975a4d31a5166e P 0bf99824798a4af795e330a314248eb1: Deleting consensus metadata
I20260429 07:59:56.660211 13458 catalog_manager.cc:5002] TS 0bf99824798a4af795e330a314248eb1 (127.12.12.129:38267): tablet 89856be9d8844f76a4975a4d31a5166e (table test-table [id=1169621ff760433c89cab2e4f82edec7]) successfully deleted
W20260429 07:59:56.741995 13964 tablet.cc:2404] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 07:59:56.759246 13993 raft_consensus.cc:493] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 07:59:56.759400 13993 raft_consensus.cc:515] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.760501 13993 leader_election.cc:290] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bf99824798a4af795e330a314248eb1 (127.12.12.129:38267), da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193)
I20260429 07:59:56.763899 13714 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.760712 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:38267 (local address 127.12.12.130:59277)
0429 07:59:56.760955 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.760970 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.761063 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.761443 (+ 380us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.761446 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.761491 (+ 45us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.761744 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.761753 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.762527 (+ 774us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.762530 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.763502 (+ 972us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.763511 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.763629 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.763647 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.763705 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.763766 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"spinlock_wait_cycles":12544}
I20260429 07:59:56.764410 13984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.760869 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:59277 (local address 127.12.12.129:38267)
0429 07:59:56.760993 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.761000 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.761014 (+ 14us) server_negotiation.cc:408] Connection header received
0429 07:59:56.761209 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.761212 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.761267 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.761334 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.761870 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.762407 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.763646 (+ 1239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.764116 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.764163 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.764215 (+ 52us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.764271 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 07:59:56.764577 14016 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.761015 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:33193 (local address 127.12.12.130:38951)
0429 07:59:56.761387 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.761405 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.761542 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.761817 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.761820 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.761841 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 07:59:56.762151 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.762160 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.762978 (+ 818us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.762985 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.763551 (+ 566us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.763558 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.764289 (+ 731us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.764302 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.764381 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.764434 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"spinlock_wait_cycles":2304,"thread_start_us":80,"threads_started":1}
I20260429 07:59:56.764617 13981 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.761186 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:38951 (local address 127.12.12.131:33193)
0429 07:59:56.761321 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.761327 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.761497 (+ 170us) server_negotiation.cc:408] Connection header received
0429 07:59:56.761656 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.761659 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.761707 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.761789 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 07:59:56.762293 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.762832 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.763714 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.764183 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.764210 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.764448 (+ 238us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.764512 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 07:59:56.765012 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a825d2566049369318c1c5f9b8958e" candidate_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf99824798a4af795e330a314248eb1" is_pre_election: true
I20260429 07:59:56.765228 13633 raft_consensus.cc:2468] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e574dc4c8a42fb98bb2333bb84fdf0 in term 0.
I20260429 07:59:56.765213 13917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a825d2566049369318c1c5f9b8958e" candidate_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8f856658b54548a7f1776107b8ffb5" is_pre_election: true
I20260429 07:59:56.765349 13917 raft_consensus.cc:2468] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e574dc4c8a42fb98bb2333bb84fdf0 in term 0.
I20260429 07:59:56.765744 13706 leader_election.cc:304] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf99824798a4af795e330a314248eb1, 55e574dc4c8a42fb98bb2333bb84fdf0; no voters:
I20260429 07:59:56.765969 13993 raft_consensus.cc:2804] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 07:59:56.766041 13993 raft_consensus.cc:493] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 07:59:56.766090 13993 raft_consensus.cc:3060] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.766963 13993 raft_consensus.cc:515] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.767230 13993 leader_election.cc:290] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [CANDIDATE]: Term 1 election: Requested vote from peers 0bf99824798a4af795e330a314248eb1 (127.12.12.129:38267), da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193)
I20260429 07:59:56.767638 13917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a825d2566049369318c1c5f9b8958e" candidate_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8f856658b54548a7f1776107b8ffb5"
I20260429 07:59:56.767638 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5a825d2566049369318c1c5f9b8958e" candidate_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf99824798a4af795e330a314248eb1"
I20260429 07:59:56.767755 13633 raft_consensus.cc:3060] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.767755 13917 raft_consensus.cc:3060] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.768792 13917 raft_consensus.cc:2468] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e574dc4c8a42fb98bb2333bb84fdf0 in term 1.
I20260429 07:59:56.768792 13633 raft_consensus.cc:2468] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e574dc4c8a42fb98bb2333bb84fdf0 in term 1.
I20260429 07:59:56.769100 13706 leader_election.cc:304] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e574dc4c8a42fb98bb2333bb84fdf0, da8f856658b54548a7f1776107b8ffb5; no voters:
I20260429 07:59:56.769301 13993 raft_consensus.cc:2804] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:56.769510 13993 raft_consensus.cc:697] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 LEADER]: Becoming Leader. State: Replica: 55e574dc4c8a42fb98bb2333bb84fdf0, State: Running, Role: LEADER
I20260429 07:59:56.769757 13993 consensus_queue.cc:237] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.772996 13473 catalog_manager.cc:5671] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130). New cstate: current_term: 1 leader_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } health_report { overall_health: UNKNOWN } } }
W20260429 07:59:56.774996 13822 tablet.cc:2404] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449595, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:56.811038 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.802526 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.12.12.190:46279)
0429 07:59:56.802665 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.802670 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.802843 (+ 173us) server_negotiation.cc:408] Connection header received
0429 07:59:56.803000 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.803002 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.803044 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.803116 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.804090 (+ 974us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.804568 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.805430 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.805607 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.807595 (+ 1988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.807616 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.807622 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.807653 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.809229 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.809798 (+ 569us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.809801 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.809803 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.809843 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.810098 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.810100 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.810101 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.810230 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.810304 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.810690 (+ 386us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.810884 (+ 194us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":59}
I20260429 07:59:56.811057 14024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.802453 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46279 (local address 127.0.0.1:44546)
0429 07:59:56.802748 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.802764 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.802876 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.803163 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.803166 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.803692 (+ 526us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:56.803965 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.803975 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.804689 (+ 714us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.804693 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.805287 (+ 594us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.805297 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.805440 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.805963 (+ 523us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:56.805979 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:56.807463 (+ 1484us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:56.809395 (+ 1932us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.809405 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.809412 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.809701 (+ 289us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.809944 (+ 243us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:56.809947 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:56.809951 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:56.810013 (+ 62us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:56.810334 (+ 321us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:56.810342 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:56.810496 (+ 154us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.810643 (+ 147us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.810813 (+ 170us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
I20260429 07:59:56.814867 13473 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:44546:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 07:59:56.815624 13473 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 07:59:56.820410 13613 tablet_service.cc:1511] Processing CreateTablet for tablet ff5fc663caf94bd6abf94bebb284a33c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0f9bcb909e34c86848944f1b74b23d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 07:59:56.820779 13613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5fc663caf94bd6abf94bebb284a33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.820675 13755 tablet_service.cc:1511] Processing CreateTablet for tablet ff5fc663caf94bd6abf94bebb284a33c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0f9bcb909e34c86848944f1b74b23d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 07:59:56.821039 13755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5fc663caf94bd6abf94bebb284a33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.822860 13987 tablet_bootstrap.cc:492] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: Bootstrap starting.
I20260429 07:59:56.823635 13988 tablet_bootstrap.cc:492] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: Bootstrap starting.
I20260429 07:59:56.823791 13987 tablet_bootstrap.cc:654] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.824023 13897 tablet_service.cc:1511] Processing CreateTablet for tablet ff5fc663caf94bd6abf94bebb284a33c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0f9bcb909e34c86848944f1b74b23d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 07:59:56.824358 13897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5fc663caf94bd6abf94bebb284a33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:56.824652 13988 tablet_bootstrap.cc:654] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.825392 13987 tablet_bootstrap.cc:492] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: No bootstrap required, opened a new log
I20260429 07:59:56.825477 13987 ts_tablet_manager.cc:1403] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 07:59:56.825814 13988 tablet_bootstrap.cc:492] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: No bootstrap required, opened a new log
I20260429 07:59:56.825870 13988 ts_tablet_manager.cc:1403] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260429 07:59:56.825945 13987 raft_consensus.cc:359] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.826042 13987 raft_consensus.cc:385] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.826071 13987 raft_consensus.cc:740] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf99824798a4af795e330a314248eb1, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.826197 13987 consensus_queue.cc:260] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.826217 13988 raft_consensus.cc:359] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.826308 13988 raft_consensus.cc:385] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.826356 13988 raft_consensus.cc:740] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e574dc4c8a42fb98bb2333bb84fdf0, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.826483 13987 ts_tablet_manager.cc:1434] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 07:59:56.826464 14008 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.826483 13988 consensus_queue.cc:260] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.826568 14008 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.826723 13988 ts_tablet_manager.cc:1434] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 07:59:56.827051 13986 tablet_bootstrap.cc:492] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: Bootstrap starting.
I20260429 07:59:56.827565 13993 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.827675 13993 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.828078 13986 tablet_bootstrap.cc:654] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:56.829293 13986 tablet_bootstrap.cc:492] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: No bootstrap required, opened a new log
I20260429 07:59:56.829406 13986 ts_tablet_manager.cc:1403] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260429 07:59:56.829900 13986 raft_consensus.cc:359] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.830018 13986 raft_consensus.cc:385] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:56.830057 13986 raft_consensus.cc:740] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8f856658b54548a7f1776107b8ffb5, State: Initialized, Role: FOLLOWER
I20260429 07:59:56.830188 13986 consensus_queue.cc:260] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.830463 13986 ts_tablet_manager.cc:1434] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 07:59:56.830482 13992 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.830596 13992 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.852614 14008 raft_consensus.cc:493] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 07:59:56.852732 14008 raft_consensus.cc:515] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.853219 14008 leader_election.cc:290] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009), da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193)
I20260429 07:59:56.853668 13760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff5fc663caf94bd6abf94bebb284a33c" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" is_pre_election: true
I20260429 07:59:56.853749 13917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff5fc663caf94bd6abf94bebb284a33c" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8f856658b54548a7f1776107b8ffb5" is_pre_election: true
I20260429 07:59:56.853824 13760 raft_consensus.cc:2468] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 0.
I20260429 07:59:56.853907 13917 raft_consensus.cc:2468] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 0.
I20260429 07:59:56.854146 13564 leader_election.cc:304] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf99824798a4af795e330a314248eb1, 55e574dc4c8a42fb98bb2333bb84fdf0; no voters:
I20260429 07:59:56.854322 14008 raft_consensus.cc:2804] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 07:59:56.854389 14008 raft_consensus.cc:493] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 07:59:56.854434 14008 raft_consensus.cc:3060] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.855296 14008 raft_consensus.cc:515] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.855644 14008 leader_election.cc:290] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 election: Requested vote from peers 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009), da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193)
I20260429 07:59:56.855931 13760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff5fc663caf94bd6abf94bebb284a33c" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0"
I20260429 07:59:56.856045 13760 raft_consensus.cc:3060] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.856029 13917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff5fc663caf94bd6abf94bebb284a33c" candidate_uuid: "0bf99824798a4af795e330a314248eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8f856658b54548a7f1776107b8ffb5"
I20260429 07:59:56.856137 13917 raft_consensus.cc:3060] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:56.856927 13760 raft_consensus.cc:2468] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 1.
I20260429 07:59:56.857021 13917 raft_consensus.cc:2468] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bf99824798a4af795e330a314248eb1 in term 1.
I20260429 07:59:56.857228 13564 leader_election.cc:304] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf99824798a4af795e330a314248eb1, 55e574dc4c8a42fb98bb2333bb84fdf0; no voters:
I20260429 07:59:56.857385 14008 raft_consensus.cc:2804] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:56.857468 14008 raft_consensus.cc:697] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf99824798a4af795e330a314248eb1, State: Running, Role: LEADER
I20260429 07:59:56.857594 14008 consensus_queue.cc:237] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } }
I20260429 07:59:56.858229 13997 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: TxnStatusTablet state changed. Reason: New leader 0bf99824798a4af795e330a314248eb1. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.858330 13997 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: This TxnStatusTablet replica's current role is: LEADER
I20260429 07:59:56.858632 14029 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 07:59:56.858803 14029 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 07:59:56.859272 13473 catalog_manager.cc:5671] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bf99824798a4af795e330a314248eb1 (127.12.12.129). New cstate: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20260429 07:59:56.870262 13984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.867412 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.12.12.129:38267)
0429 07:59:56.867531 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.867534 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.867636 (+ 102us) server_negotiation.cc:408] Connection header received
0429 07:59:56.867791 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.867794 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.867837 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.867902 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 07:59:56.868297 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.868738 (+ 441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.869659 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.869839 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.869876 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.869938 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.869999 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.870132 (+ 133us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.870166 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 07:59:56.870316 14024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.867327 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:38267 (local address 127.0.0.1:60908)
0429 07:59:56.867562 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.867578 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.867695 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.867940 (+ 245us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.867943 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.867952 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 07:59:56.868186 (+ 234us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.868193 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.868869 (+ 676us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.868874 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.869531 (+ 657us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.869540 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.869674 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.869714 (+ 40us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.870032 (+ 318us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.870043 (+ 11us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.870116 (+ 73us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.870184 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260429 07:59:56.872614 13917 raft_consensus.cc:1275] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Refusing update from remote peer 0bf99824798a4af795e330a314248eb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 07:59:56.872685 13760 raft_consensus.cc:1275] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Refusing update from remote peer 0bf99824798a4af795e330a314248eb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 07:59:56.873039 14008 consensus_queue.cc:1048] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 07:59:56.873162 13997 consensus_queue.cc:1048] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 07:59:56.874660 13992 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: TxnStatusTablet state changed. Reason: New leader 0bf99824798a4af795e330a314248eb1. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.874684 13993 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: TxnStatusTablet state changed. Reason: New leader 0bf99824798a4af795e330a314248eb1. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.874763 13993 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.874764 13992 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.875691 13994 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.875778 13994 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: This TxnStatusTablet replica's current role is: LEADER
I20260429 07:59:56.875854 14000 mvcc.cc:204] Tried to move back new op lower bound from 7280433548786343936 to 7280433548730023936. Current Snapshot: MvccSnapshot[applied={T|T < 7280433548786343936}]
I20260429 07:59:56.876680 13993 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.876784 13993 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 55e574dc4c8a42fb98bb2333bb84fdf0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.879593 14008 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.879698 14008 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P 0bf99824798a4af795e330a314248eb1: This TxnStatusTablet replica's current role is: LEADER
I20260429 07:59:56.880247 13992 tablet_replica.cc:442] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bf99824798a4af795e330a314248eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 } } peers { permanent_uuid: "55e574dc4c8a42fb98bb2333bb84fdf0" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46009 } } peers { permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 } } }
I20260429 07:59:56.880359 13992 tablet_replica.cc:445] T ff5fc663caf94bd6abf94bebb284a33c P da8f856658b54548a7f1776107b8ffb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 07:59:56.885735 13985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.882470 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.12.12.130:46009)
0429 07:59:56.882593 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.882615 (+ 22us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.882628 (+ 13us) server_negotiation.cc:408] Connection header received
0429 07:59:56.882703 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.882707 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.882767 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.882886 (+ 119us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 07:59:56.883317 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.884053 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.884658 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.884889 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.884945 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.885383 (+ 438us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.885479 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.885571 (+ 92us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.885615 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 07:59:56.895665 14035 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.892188 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:38267 (local address 127.12.12.130:38581)
0429 07:59:56.892572 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.892589 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.892681 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.892934 (+ 253us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.892936 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.892942 (+ 6us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 07:59:56.893137 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.893141 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.894062 (+ 921us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.894064 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.894945 (+ 881us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.894953 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.895040 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.895068 (+ 28us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.895457 (+ 389us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.895460 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.895511 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.895557 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260429 07:59:56.895700 13984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.892324 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:38581 (local address 127.12.12.129:38267)
0429 07:59:56.892475 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.892479 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.892641 (+ 162us) server_negotiation.cc:408] Connection header received
0429 07:59:56.892764 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.892767 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.892846 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.892924 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 07:59:56.893239 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.893951 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.895056 (+ 1105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.895240 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.895279 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.895353 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.895445 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.895545 (+ 100us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.895582 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 07:59:56.902417 13917 raft_consensus.cc:1275] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Refusing update from remote peer 55e574dc4c8a42fb98bb2333bb84fdf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 07:59:56.902650 13633 raft_consensus.cc:1275] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Refusing update from remote peer 55e574dc4c8a42fb98bb2333bb84fdf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 07:59:56.902863 14017 consensus_queue.cc:1048] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8f856658b54548a7f1776107b8ffb5" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 07:59:56.903107 13993 consensus_queue.cc:1048] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf99824798a4af795e330a314248eb1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 38267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 07:59:56.925420 14039 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.921807 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46009 (local address 127.12.12.129:38391)
0429 07:59:56.922122 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:56.922137 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:56.922267 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:56.922519 (+ 252us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:56.922522 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:56.922529 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 07:59:56.922876 (+ 347us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.922885 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.923882 (+ 997us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.923886 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:56.924436 (+ 550us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:56.924445 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.924577 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.924658 (+ 81us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.925161 (+ 503us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.925167 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 07:59:56.925233 (+ 66us) client_negotiation.cc:241] Negotiation successful
0429 07:59:56.925297 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260429 07:59:56.925403 13985 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.921840 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:38391 (local address 127.12.12.130:46009)
0429 07:59:56.921951 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.921955 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.922217 (+ 262us) server_negotiation.cc:408] Connection header received
0429 07:59:56.922362 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.922365 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.922418 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.922487 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 07:59:56.923033 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.923771 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.924566 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.924769 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.924953 (+ 184us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 07:59:56.925061 (+ 108us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 07:59:56.925164 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.925241 (+ 77us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.925276 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 07:59:56.948663 13472 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 07:59:56.948863 13472 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 07:59:56.951273 13472 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 092d7feac12a46e7b485d671d3eb737d: Sending DeleteTablet for 3 replicas of tablet e5a825d2566049369318c1c5f9b8958e
I20260429 07:59:56.951996 13613 tablet_service.cc:1558] Processing DeleteTablet for tablet e5a825d2566049369318c1c5f9b8958e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 07:59:56 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:60890
I20260429 07:59:56.952127 13897 tablet_service.cc:1558] Processing DeleteTablet for tablet e5a825d2566049369318c1c5f9b8958e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 07:59:56 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:35358
I20260429 07:59:56.952183 14012 tablet_replica.cc:333] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: stopping tablet replica
I20260429 07:59:56.952298 14012 raft_consensus.cc:2243] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 07:59:56.952320 14010 tablet_replica.cc:333] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: stopping tablet replica
I20260429 07:59:56.952461 14012 raft_consensus.cc:2272] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 07:59:56.952456 14010 raft_consensus.cc:2243] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 07:59:56.952575 14010 raft_consensus.cc:2272] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 07:59:56.952991 13755 tablet_service.cc:1558] Processing DeleteTablet for tablet e5a825d2566049369318c1c5f9b8958e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 07:59:56 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49326
I20260429 07:59:56.953182 14011 tablet_replica.cc:333] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: stopping tablet replica
I20260429 07:59:56.953313 14011 raft_consensus.cc:2243] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 LEADER]: Raft consensus shutting down.
I20260429 07:59:56.953606 14011 raft_consensus.cc:2272] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 07:59:56.953625 14010 ts_tablet_manager.cc:1916] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 07:59:56.954315 14011 ts_tablet_manager.cc:1916] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 07:59:56.954921 14012 ts_tablet_manager.cc:1916] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 07:59:56.957350 14012 ts_tablet_manager.cc:1929] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 07:59:56.957350 14010 ts_tablet_manager.cc:1929] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 07:59:56.957427 14012 log.cc:1199] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/e5a825d2566049369318c1c5f9b8958e
I20260429 07:59:56.957427 14010 log.cc:1199] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/e5a825d2566049369318c1c5f9b8958e
I20260429 07:59:56.957682 14010 ts_tablet_manager.cc:1950] T e5a825d2566049369318c1c5f9b8958e P da8f856658b54548a7f1776107b8ffb5: Deleting consensus metadata
I20260429 07:59:56.957682 14012 ts_tablet_manager.cc:1950] T e5a825d2566049369318c1c5f9b8958e P 0bf99824798a4af795e330a314248eb1: Deleting consensus metadata
I20260429 07:59:56.957912 14011 ts_tablet_manager.cc:1929] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 07:59:56.958030 14011 log.cc:1199] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/e5a825d2566049369318c1c5f9b8958e
I20260429 07:59:56.958336 14011 ts_tablet_manager.cc:1950] T e5a825d2566049369318c1c5f9b8958e P 55e574dc4c8a42fb98bb2333bb84fdf0: Deleting consensus metadata
I20260429 07:59:56.958680 13458 catalog_manager.cc:5002] TS 0bf99824798a4af795e330a314248eb1 (127.12.12.129:38267): tablet e5a825d2566049369318c1c5f9b8958e (table test-table [id=0e4b0c7c4ff742f889595173e50a2148]) successfully deleted
I20260429 07:59:56.959049 13458 catalog_manager.cc:5002] TS da8f856658b54548a7f1776107b8ffb5 (127.12.12.131:33193): tablet e5a825d2566049369318c1c5f9b8958e (table test-table [id=0e4b0c7c4ff742f889595173e50a2148]) successfully deleted
I20260429 07:59:56.959985 13458 catalog_manager.cc:5002] TS 55e574dc4c8a42fb98bb2333bb84fdf0 (127.12.12.130:46009): tablet e5a825d2566049369318c1c5f9b8958e (table test-table [id=0e4b0c7c4ff742f889595173e50a2148]) successfully deleted
Apr 29 07:59:56 dist-test-slave-nhqz krb5kdc[13426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 07:59:56.968568 13984 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.959278 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.12.12.129:38267)
0429 07:59:56.959437 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.959441 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.960019 (+ 578us) server_negotiation.cc:408] Connection header received
0429 07:59:56.960236 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.960240 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.960295 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.960346 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.961299 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.961792 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.962445 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.962992 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.965211 (+ 2219us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.965240 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.965252 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.965274 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.966894 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.967316 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.967321 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.967322 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.967398 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.967634 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.967638 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.967640 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.967805 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.967996 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.968206 (+ 210us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.968326 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260429 07:59:56.969035 13673 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60922
I20260429 07:59:56.977499 13523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:56.970679 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.12.12.190:46279)
0429 07:59:56.970823 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:56.970827 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:56.971221 (+ 394us) server_negotiation.cc:408] Connection header received
0429 07:59:56.971352 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:56.971355 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:56.971415 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:56.971497 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:56.972164 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.972647 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:56.973288 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:56.973788 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:56.974450 (+ 662us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:56.974469 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:56.974474 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:56.974502 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:56.975987 (+ 1485us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.976388 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.976392 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.976393 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.976436 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:56.976666 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:56.976669 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:56.976671 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:56.976862 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:56.977008 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:56.977215 (+ 207us) server_negotiation.cc:300] Negotiation successful
0429 07:59:56.977321 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260429 07:59:56.978068 13472 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554
I20260429 07:59:56.979264 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 13548
I20260429 07:59:56.987488 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 13690
I20260429 07:59:56.994266 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 13832
I20260429 07:59:57.000492 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 13442
2026-04-29T07:59:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3464 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 07:59:57 dist-test-slave-nhqz krb5kdc[14050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 07:59:57 dist-test-slave-nhqz krb5kdc[14050](info): set up 2 sockets
Apr 29 07:59:57 dist-test-slave-nhqz krb5kdc[14050](info): commencing operation
krb5kdc: starting...
W20260429 07:59:58.974012 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.951s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T07:59:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T07:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:59.109735 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:41491
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:41491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:59.211755 14066 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:59.211966 14066 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:59.212018 14066 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:59.215270 14066 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 07:59:59.215328 14066 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 07:59:59.215346 14066 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:59.215415 14066 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:59.215449 14066 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 07:59:59.215494 14066 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 07:59:59.219853 14066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:41491
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:41491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:59.220923 14066 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:59.221704 14066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:59.227226 14072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.227231 14071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.227234 14074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:59.227723 14066 server_base.cc:1061] running on GCE node
I20260429 07:59:59.228199 14066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:59.229075 14066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:59.230283 14066 hybrid_clock.cc:648] HybridClock initialized: now 1777449599230274 us; error 39 us; skew 500 ppm
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:59.232901 14066 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 07:59:59.233896 14066 webserver.cc:492] Webserver started at http://127.12.12.190:35095/ using document root <none> and password file <none>
I20260429 07:59:59.234444 14066 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:59.234570 14066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:59.234778 14066 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:59.236444 14066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "ab9e7df8a02348eca3566c1a02f2bae5"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "75eebe184aba4df98e9aef066f9a81a0"
server_key_iv: "a0c4ddb2cd911f8f1abc493cf974e52d"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.236976 14066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "ab9e7df8a02348eca3566c1a02f2bae5"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "75eebe184aba4df98e9aef066f9a81a0"
server_key_iv: "a0c4ddb2cd911f8f1abc493cf974e52d"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.239928 14066 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 07:59:59.241926 14081 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.242820 14066 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 07:59:59.242971 14066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "ab9e7df8a02348eca3566c1a02f2bae5"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "75eebe184aba4df98e9aef066f9a81a0"
server_key_iv: "a0c4ddb2cd911f8f1abc493cf974e52d"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.243068 14066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:59.258661 14066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:59.264472 14066 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:59.264668 14066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:59.272274 14066 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:41491
I20260429 07:59:59.272282 14143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:41491 every 8 connection(s)
I20260429 07:59:59.273296 14066 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 07:59:59.275509 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 14066
I20260429 07:59:59.275643 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 07:59:59.275907 12338 external_mini_cluster.cc:1468] Setting key 5fc49432609067d3a4b0c52c45b0ab8a
I20260429 07:59:59.276454 14144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 07:59:59.281410 14144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Bootstrap starting.
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:59.283627 14144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Neither blocks nor log segments found. Creating new log.
I20260429 07:59:59.284423 14144 log.cc:826] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Log is configured to *not* fsync() on all Append() calls
I20260429 07:59:59.286144 14144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: No bootstrap required, opened a new log
I20260429 07:59:59.287985 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.277150 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.12.12.190:41491)
0429 07:59:59.277575 (+ 425us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.277584 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.277615 (+ 31us) server_negotiation.cc:408] Connection header received
0429 07:59:59.278226 (+ 611us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.278241 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.278509 (+ 268us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.278813 (+ 304us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.279707 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.280441 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.281164 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.281442 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.283700 (+ 2258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.283737 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.283752 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.283791 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.285630 (+ 1839us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.286038 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.286043 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.286047 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.286110 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.286390 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.286394 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.286395 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.286664 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.286865 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.287137 (+ 272us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.287343 (+ 206us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":143,"threads_started":1}
I20260429 07:59:59.288537 14144 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41491 } }
I20260429 07:59:59.288750 14144 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 07:59:59.288885 14144 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9e7df8a02348eca3566c1a02f2bae5, State: Initialized, Role: FOLLOWER
I20260429 07:59:59.289289 14144 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41491 } }
I20260429 07:59:59.289430 14144 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 07:59:59.289510 14144 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 07:59:59.289624 14144 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 0 FOLLOWER]: Advancing to term 1
I20260429 07:59:59.290504 14144 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41491 } }
I20260429 07:59:59.290809 14144 leader_election.cc:304] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9e7df8a02348eca3566c1a02f2bae5; no voters:
I20260429 07:59:59.291106 14144 leader_election.cc:290] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 07:59:59.291194 14149 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 07:59:59.291411 14149 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [term 1 LEADER]: Becoming Leader. State: Replica: ab9e7df8a02348eca3566c1a02f2bae5, State: Running, Role: LEADER
I20260429 07:59:59.291745 14149 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41491 } }
I20260429 07:59:59.292162 14144 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 07:59:59.292788 14151 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9e7df8a02348eca3566c1a02f2bae5. Latest consensus state: current_term: 1 leader_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41491 } } }
I20260429 07:59:59.292933 14151 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:59.293196 14150 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e7df8a02348eca3566c1a02f2bae5" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41491 } } }
I20260429 07:59:59.293300 14150 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5 [sys.catalog]: This master's current role is: LEADER
I20260429 07:59:59.293542 14156 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 07:59:59.296414 14156 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 07:59:59.301333 14156 catalog_manager.cc:1357] Generated new cluster ID: f0de05d1bd1e43cdbbaf6fd764dc9dbe
I20260429 07:59:59.301389 14156 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 07:59:59.333760 14156 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 07:59:59.334751 14156 catalog_manager.cc:1514] Loading token signing keys...
I20260429 07:59:59.348290 14156 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Generated new TSK 0
I20260429 07:59:59.349050 14156 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:59.404011 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:41491
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:59.503881 14172 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:59.504101 14172 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:59.504163 14172 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:59.507982 14172 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:59.508041 14172 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:59.508122 14172 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 07:59:59.508152 14172 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 07:59:59.508222 14172 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 07:59:59.512368 14172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:41491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:59.513377 14172 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:59.514175 14172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 07:59:59.520182 14177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.520191 14178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.520211 14180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:59.520447 14172 server_base.cc:1061] running on GCE node
I20260429 07:59:59.520774 14172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:59.521337 14172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:59.522506 14172 hybrid_clock.cc:648] HybridClock initialized: now 1777449599522494 us; error 32 us; skew 500 ppm
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:59.525131 14172 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 07:59:59.526213 14172 webserver.cc:492] Webserver started at http://127.12.12.129:46329/ using document root <none> and password file <none>
I20260429 07:59:59.526737 14172 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:59.526808 14172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:59.527035 14172 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:59.528708 14172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "9053f40d69cc4260a8218eaa12684cbf"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "8e11a7d11abaf4930c14d0bb35815c14"
server_key_iv: "2f3ee48063c76f2a99d8ebe5120d908e"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.529286 14172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "9053f40d69cc4260a8218eaa12684cbf"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "8e11a7d11abaf4930c14d0bb35815c14"
server_key_iv: "2f3ee48063c76f2a99d8ebe5120d908e"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.532444 14172 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 07:59:59.534386 14187 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.535203 14172 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 07:59:59.535323 14172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "9053f40d69cc4260a8218eaa12684cbf"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "8e11a7d11abaf4930c14d0bb35815c14"
server_key_iv: "2f3ee48063c76f2a99d8ebe5120d908e"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.535413 14172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:59.548568 14172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:59.550906 14172 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:59.551129 14172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:59.552476 14172 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:59.552556 14172 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.552603 14172 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:59.552629 14172 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:59.563985 14302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:45287 every 8 connection(s)
I20260429 07:59:59.563966 14172 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:45287
I20260429 07:59:59.564530 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.552988 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:39905 (local address 127.12.12.190:41491)
0429 07:59:59.553113 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.553117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.553800 (+ 683us) server_negotiation.cc:408] Connection header received
0429 07:59:59.554637 (+ 837us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.554641 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.554701 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.554801 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.556172 (+ 1371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.556848 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.557811 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.558048 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.560650 (+ 2602us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.560669 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.560672 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.560697 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.562273 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.562861 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.562866 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.562872 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.562945 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.563297 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.563300 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.563302 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.563481 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.563636 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.564203 (+ 567us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.564348 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 07:59:59.565209 14172 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 07:59:59.565207 14196 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.553291 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.12.12.129:39905)
0429 07:59:59.553684 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:59.553711 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:59.554428 (+ 717us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:59.554944 (+ 516us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:59.554951 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:59.555384 (+ 433us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:59.555983 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.555997 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.556957 (+ 960us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.556963 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:59.557642 (+ 679us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.557654 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.557879 (+ 225us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.558465 (+ 586us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:59.558490 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:59.560487 (+ 1997us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:59.562370 (+ 1883us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.562379 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.562393 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.562763 (+ 370us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.563043 (+ 280us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.563046 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.563048 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.563154 (+ 106us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.563635 (+ 481us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:59.563642 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:59.563919 (+ 277us) client_negotiation.cc:770] Sending connection context
0429 07:59:59.564148 (+ 229us) client_negotiation.cc:241] Negotiation successful
0429 07:59:59.564396 (+ 248us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260429 07:59:59.566970 14303 heartbeater.cc:344] Connected to a master server at 127.12.12.190:41491
I20260429 07:59:59.567135 14303 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:59.567757 14303 heartbeater.cc:507] Master 127.12.12.190:41491 requested a full tablet report, sending...
I20260429 07:59:59.569289 14098 ts_manager.cc:194] Registered new tserver with Master: 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287)
I20260429 07:59:59.569393 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 14172
I20260429 07:59:59.569522 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 07:59:59.569830 12338 external_mini_cluster.cc:1468] Setting key a43b8dfb3090deb9263efa911fab763e
I20260429 07:59:59.570612 14098 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:39905
I20260429 07:59:59.576218 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.568908 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:49777 (local address 127.12.12.190:41491)
0429 07:59:59.569062 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.569069 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.569218 (+ 149us) server_negotiation.cc:408] Connection header received
0429 07:59:59.569341 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.569346 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.569400 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.569498 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.570238 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.570943 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.571751 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.572295 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.572875 (+ 580us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.572894 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.572901 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.572931 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.574714 (+ 1783us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.575180 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.575184 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.575187 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.575234 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.575488 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.575492 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.575494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.575630 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.575737 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.575975 (+ 238us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.576087 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 07:59:59.576242 14309 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.568805 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.12.12.129:49777)
0429 07:59:59.569120 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:59.569136 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:59.569244 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:59.569616 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:59.569620 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:59.569829 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:59.570100 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.570109 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.571064 (+ 955us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.571068 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:59.571609 (+ 541us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.571619 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.571737 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.572308 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:59.572327 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:59.572722 (+ 395us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:59.574847 (+ 2125us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.574851 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.574852 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.575028 (+ 176us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.575338 (+ 310us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.575341 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.575344 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.575397 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.575727 (+ 330us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:59.575730 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:59.575817 (+ 87us) client_negotiation.cc:770] Sending connection context
0429 07:59:59.575924 (+ 107us) client_negotiation.cc:241] Negotiation successful
0429 07:59:59.576049 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:59.616496 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:41491
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:59.719698 14314 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:59.719951 14314 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:59.720077 14314 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:59.723589 14314 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:59.723654 14314 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:59.723739 14314 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 07:59:59.723775 14314 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 07:59:59.723850 14314 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 07:59:59.728173 14314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:41491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:59.729271 14314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:59.730088 14314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 07:59:59.736610 14314 server_base.cc:1061] running on GCE node
W20260429 07:59:59.736536 14319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.736510 14322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.736549 14320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:59.737079 14314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:59.737656 14314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:59.738821 14314 hybrid_clock.cc:648] HybridClock initialized: now 1777449599738800 us; error 31 us; skew 500 ppm
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:59.741384 14314 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 07:59:59.742313 14314 webserver.cc:492] Webserver started at http://127.12.12.130:42119/ using document root <none> and password file <none>
I20260429 07:59:59.742812 14314 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:59.742897 14314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:59.743117 14314 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:59.744704 14314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "121d0ed40de3420ea5950c5ec34fada8"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "acbf40fb2d82d2c6def2bb6cfab42233"
server_key_iv: "cbfebb775b8ee638bd5e5d3fee521fdf"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.745222 14314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "121d0ed40de3420ea5950c5ec34fada8"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "acbf40fb2d82d2c6def2bb6cfab42233"
server_key_iv: "cbfebb775b8ee638bd5e5d3fee521fdf"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.748297 14314 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 07:59:59.750306 14329 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.751223 14314 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 07:59:59.751351 14314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "121d0ed40de3420ea5950c5ec34fada8"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "acbf40fb2d82d2c6def2bb6cfab42233"
server_key_iv: "cbfebb775b8ee638bd5e5d3fee521fdf"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.751442 14314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:59.762506 14314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:59.765230 14314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:59.765372 14314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:59.766749 14314 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:59.766829 14314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.766878 14314 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:59.766906 14314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:59.778126 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.767244 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:52375 (local address 127.12.12.190:41491)
0429 07:59:59.767382 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.767386 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.768157 (+ 771us) server_negotiation.cc:408] Connection header received
0429 07:59:59.769015 (+ 858us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.769019 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.769085 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.769174 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.770485 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.771114 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.771844 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.772041 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.774477 (+ 2436us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.774500 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.774502 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.774526 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.776222 (+ 1696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.776759 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.776763 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.776767 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.776857 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.777187 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.777190 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.777192 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.777329 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.777432 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.777852 (+ 420us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.777961 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 07:59:59.778501 14314 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:45961
I20260429 07:59:59.778522 14444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:45961 every 8 connection(s)
I20260429 07:59:59.778750 14338 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.767602 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.12.12.130:52375)
0429 07:59:59.768024 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:59.768054 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:59.768773 (+ 719us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:59.769329 (+ 556us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:59.769336 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:59.769832 (+ 496us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:59.770345 (+ 513us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.770356 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.771227 (+ 871us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.771231 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:59.771737 (+ 506us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.771747 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.771936 (+ 189us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.772641 (+ 705us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:59.772657 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:59.774295 (+ 1638us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:59.776357 (+ 2062us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.776363 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.776391 (+ 28us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.776653 (+ 262us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.776975 (+ 322us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.776978 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.776981 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.777071 (+ 90us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.777433 (+ 362us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:59.777439 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:59.777635 (+ 196us) client_negotiation.cc:770] Sending connection context
0429 07:59:59.777820 (+ 185us) client_negotiation.cc:241] Negotiation successful
0429 07:59:59.778013 (+ 193us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260429 07:59:59.779466 14314 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 07:59:59.781865 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 14314
I20260429 07:59:59.781991 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 07:59:59.782279 12338 external_mini_cluster.cc:1468] Setting key 86956ad107a8f8ecf4d89146d09e0819
I20260429 07:59:59.782755 14445 heartbeater.cc:344] Connected to a master server at 127.12.12.190:41491
I20260429 07:59:59.782943 14445 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:59.783442 14445 heartbeater.cc:507] Master 127.12.12.190:41491 requested a full tablet report, sending...
I20260429 07:59:59.784392 14097 ts_manager.cc:194] Registered new tserver with Master: 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961)
I20260429 07:59:59.785343 14097 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:52375
I20260429 07:59:59.788947 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.782289 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:34311 (local address 127.12.12.190:41491)
0429 07:59:59.782404 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.782408 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.782611 (+ 203us) server_negotiation.cc:408] Connection header received
0429 07:59:59.782727 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.782730 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.782776 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.782851 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.783441 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.784002 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.784873 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.785051 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.785900 (+ 849us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.785912 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.785914 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.785934 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.787308 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.787753 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.787758 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.787761 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.787820 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.788120 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.788122 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.788123 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.788272 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.788366 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.788676 (+ 310us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.788759 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 07:59:59.789031 14451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.782194 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.12.12.130:34311)
0429 07:59:59.782521 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:59.782537 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:59.782634 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:59.782967 (+ 333us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:59.782970 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:59.783134 (+ 164us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:59.783320 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.783325 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.784117 (+ 792us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.784121 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:59.784710 (+ 589us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.784722 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.784879 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.785364 (+ 485us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:59.785385 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:59.785788 (+ 403us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:59.787438 (+ 1650us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.787442 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.787445 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.787660 (+ 215us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.787929 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.787934 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.787939 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.788003 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.788398 (+ 395us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:59.788416 (+ 18us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:59.788518 (+ 102us) client_negotiation.cc:770] Sending connection context
0429 07:59:59.788632 (+ 114us) client_negotiation.cc:241] Negotiation successful
0429 07:59:59.788761 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 07:59:59.828411 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:41491
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 07:59:59.927541 14456 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 07:59:59.927752 14456 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 07:59:59.927812 14456 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 07:59:59.931072 14456 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 07:59:59.931143 14456 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 07:59:59.931195 14456 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 07:59:59.931214 14456 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 07:59:59.931253 14456 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 07:59:59.935338 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:41491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 07:59:59.936363 14456 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 07:59:59.937268 14456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 07:59:59.943585 14456 server_base.cc:1061] running on GCE node
W20260429 07:59:59.943491 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.943468 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 07:59:59.943466 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 07:59:59.944031 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 07:59:59.944592 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 07:59:59.945744 14456 hybrid_clock.cc:648] HybridClock initialized: now 1777449599945716 us; error 37 us; skew 500 ppm
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 07:59:59.948204 14456 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 07:59:59.949172 14456 webserver.cc:492] Webserver started at http://127.12.12.131:39395/ using document root <none> and password file <none>
I20260429 07:59:59.949704 14456 fs_manager.cc:362] Metadata directory not provided
I20260429 07:59:59.949780 14456 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 07:59:59.949985 14456 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 07:59:59.951589 14456 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "1e3aaf4d86b744449e1df19a35e8722e"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "9a866983abccc1aaf7cc9a61f7635da3"
server_key_iv: "f7c405123dd775a75d50ea6ed3fa2b79"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.952069 14456 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "1e3aaf4d86b744449e1df19a35e8722e"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "9a866983abccc1aaf7cc9a61f7635da3"
server_key_iv: "f7c405123dd775a75d50ea6ed3fa2b79"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.955346 14456 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260429 07:59:59.957373 14471 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.958305 14456 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 07:59:59.958454 14456 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "1e3aaf4d86b744449e1df19a35e8722e"
format_stamp: "Formatted at 2026-04-29 07:59:59 on dist-test-slave-nhqz"
server_key: "9a866983abccc1aaf7cc9a61f7635da3"
server_key_iv: "f7c405123dd775a75d50ea6ed3fa2b79"
server_key_version: "encryptionkey@0"
I20260429 07:59:59.958549 14456 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 07:59:59.970145 14456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 07:59:59.972273 14456 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 07:59:59.972457 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 07:59:59.973848 14456 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 07:59:59.973924 14456 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 07:59:59.974025 14456 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 07:59:59.974077 14456 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 07:59:59 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 07:59:59.985070 14456 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:44817
I20260429 07:59:59.985118 14586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:44817 every 8 connection(s)
I20260429 07:59:59.985781 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.974421 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:39139 (local address 127.12.12.190:41491)
0429 07:59:59.974589 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.974594 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.975285 (+ 691us) server_negotiation.cc:408] Connection header received
0429 07:59:59.976193 (+ 908us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.976198 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.976267 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.976372 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.977886 (+ 1514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.978623 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.979328 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.979502 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.982023 (+ 2521us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.982049 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.982051 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.982078 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.983785 (+ 1707us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.984307 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.984311 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.984312 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.984366 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.984679 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.984686 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.984690 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.984913 (+ 223us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.985036 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.985511 (+ 475us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.985622 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 07:59:59.986095 14456 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 07:59:59.986244 14480 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.974703 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.12.12.131:39139)
0429 07:59:59.975131 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:59.975167 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:59.975911 (+ 744us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:59.976532 (+ 621us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:59.976539 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:59.977053 (+ 514us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:59.977734 (+ 681us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.977750 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.978809 (+ 1059us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.978813 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:59.979208 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.979215 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.979417 (+ 202us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.979999 (+ 582us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:59.980025 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:59.981877 (+ 1852us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:59.983910 (+ 2033us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.983916 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.983930 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.984206 (+ 276us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.984457 (+ 251us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.984459 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.984460 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.984550 (+ 90us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.985014 (+ 464us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:59.985038 (+ 24us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:59.985287 (+ 249us) client_negotiation.cc:770] Sending connection context
0429 07:59:59.985440 (+ 153us) client_negotiation.cc:241] Negotiation successful
0429 07:59:59.985595 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":126,"threads_started":1}
I20260429 07:59:59.987767 14587 heartbeater.cc:344] Connected to a master server at 127.12.12.190:41491
I20260429 07:59:59.987922 14587 heartbeater.cc:461] Registering TS with master...
I20260429 07:59:59.988457 14587 heartbeater.cc:507] Master 127.12.12.190:41491 requested a full tablet report, sending...
I20260429 07:59:59.989470 14097 ts_manager.cc:194] Registered new tserver with Master: 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 07:59:59.990236 14097 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:39139
I20260429 07:59:59.993327 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 14456
I20260429 07:59:59.993571 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 07:59:59.993847 12338 external_mini_cluster.cc:1468] Setting key b0ac43a981e6eb80dde6b04bdd497789
I20260429 07:59:59.995777 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.989505 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:33447 (local address 127.12.12.190:41491)
0429 07:59:59.989622 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0429 07:59:59.989625 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 07:59:59.989731 (+ 106us) server_negotiation.cc:408] Connection header received
0429 07:59:59.989823 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 07:59:59.989826 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 07:59:59.989879 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 07:59:59.989943 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 07:59:59.990529 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.991002 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.991739 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.991854 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.992652 (+ 798us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 07:59:59.992664 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 07:59:59.992667 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 07:59:59.992687 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0429 07:59:59.994185 (+ 1498us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.994668 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.994673 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.994674 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.994727 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 07:59:59.994995 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 07:59:59.995001 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 07:59:59.995006 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 07:59:59.995179 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 07:59:59.995290 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 07:59:59.995536 (+ 246us) server_negotiation.cc:300] Negotiation successful
0429 07:59:59.995638 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 07:59:59.995813 14593 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 07:59:59.989369 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.12.12.131:33447)
0429 07:59:59.989651 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0429 07:59:59.989665 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 07:59:59.989752 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 07:59:59.989985 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 07:59:59.989988 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 07:59:59.990172 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0429 07:59:59.990408 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.990415 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.991133 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 07:59:59.991137 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 07:59:59.991632 (+ 495us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 07:59:59.991640 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 07:59:59.991749 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 07:59:59.992284 (+ 535us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 07:59:59.992297 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 07:59:59.992545 (+ 248us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 07:59:59.994301 (+ 1756us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.994304 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.994306 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.994538 (+ 232us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.994837 (+ 299us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 07:59:59.994840 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 07:59:59.994842 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 07:59:59.994897 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 07:59:59.995286 (+ 389us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 07:59:59.995289 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 07:59:59.995352 (+ 63us) client_negotiation.cc:770] Sending connection context
0429 07:59:59.995464 (+ 112us) client_negotiation.cc:241] Negotiation successful
0429 07:59:59.995552 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260429 07:59:59.996064 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 08:00:00 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 08:00:00 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:00.020053 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.012339 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.12.12.190:41491)
0429 08:00:00.012554 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.012566 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.012580 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:00:00.012622 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.012625 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.012671 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.012788 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:00.013720 (+ 932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.014268 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.014969 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.015184 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.017492 (+ 2308us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:00.017510 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:00.017513 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:00.017540 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:00.018792 (+ 1252us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:00.019150 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:00.019152 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:00.019153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:00.019193 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:00.019376 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:00.019378 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:00.019379 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:00.019509 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:00.019606 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.019784 (+ 178us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.019912 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260429 08:00:00.022778 14097 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:00.025306 14097 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:00.037739 14604 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.033044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:44817 (local address 127.0.0.1:41564)
0429 08:00:00.033640 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.033674 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.034066 (+ 392us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.034956 (+ 890us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.034962 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.034984 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.035250 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.035256 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.036303 (+ 1047us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.036308 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.037241 (+ 933us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.037250 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.037356 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.037420 (+ 64us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.037504 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.037574 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":105,"threads_started":1}
I20260429 08:00:00.037744 14607 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.033328 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:45287 (local address 127.0.0.1:44160)
0429 08:00:00.033919 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.033933 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.034054 (+ 121us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.035035 (+ 981us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.035041 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.035054 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.035256 (+ 202us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.035262 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.036537 (+ 1275us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.036543 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.037257 (+ 714us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.037265 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.037375 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.037419 (+ 44us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.037509 (+ 90us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.037582 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":92,"threads_started":1}
I20260429 08:00:00.038401 14606 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.033678 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.0.0.1:55976)
0429 08:00:00.034243 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.034256 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.034323 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.034803 (+ 480us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.034808 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.034834 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.035119 (+ 285us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.035126 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.036488 (+ 1362us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.036491 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.037329 (+ 838us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.037335 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.037415 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.037424 (+ 9us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.037504 (+ 80us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.037575 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":121,"threads_started":1}
I20260429 08:00:00.038776 14608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.033784 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.12.12.129:45287)
0429 08:00:00.034537 (+ 753us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.034547 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.034564 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:00:00.034625 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.034628 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.034776 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.034900 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.035373 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.036432 (+ 1059us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.037832 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.038382 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.038504 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.038580 (+ 76us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.038665 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":108,"threads_started":1}
I20260429 08:00:00.039106 14609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.033822 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.12.12.130:45961)
0429 08:00:00.034332 (+ 510us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.034340 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.034375 (+ 35us) server_negotiation.cc:408] Connection header received
0429 08:00:00.034461 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.034466 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.034642 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.034776 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.035235 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.036380 (+ 1145us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.037665 (+ 1285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.038710 (+ 1045us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.038815 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.038875 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.038962 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":94,"threads_started":1}
I20260429 08:00:00.039178 14605 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.033183 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.12.12.131:44817)
0429 08:00:00.034569 (+ 1386us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.034575 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.034592 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:00:00.034664 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.034669 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.034810 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.035393 (+ 583us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.035513 (+ 120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.036185 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.038300 (+ 2115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.038849 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.038925 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.038978 (+ 53us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.039044 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":117,"threads_started":1}
I20260429 08:00:00.041673 14379 tablet_service.cc:1511] Processing CreateTablet for tablet 16f0a7f20b3d4453b3db0d45317f8c48 (DEFAULT_TABLE table=test-table [id=bcf242d6c5e640a8ba40bfa6874d2ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:00.041679 14237 tablet_service.cc:1511] Processing CreateTablet for tablet 16f0a7f20b3d4453b3db0d45317f8c48 (DEFAULT_TABLE table=test-table [id=bcf242d6c5e640a8ba40bfa6874d2ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:00.041679 14521 tablet_service.cc:1511] Processing CreateTablet for tablet 16f0a7f20b3d4453b3db0d45317f8c48 (DEFAULT_TABLE table=test-table [id=bcf242d6c5e640a8ba40bfa6874d2ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:00.042508 14379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f0a7f20b3d4453b3db0d45317f8c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.042518 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f0a7f20b3d4453b3db0d45317f8c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.042518 14521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f0a7f20b3d4453b3db0d45317f8c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.047746 14610 tablet_bootstrap.cc:492] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Bootstrap starting.
I20260429 08:00:00.048792 14611 tablet_bootstrap.cc:492] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Bootstrap starting.
I20260429 08:00:00.048990 14612 tablet_bootstrap.cc:492] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Bootstrap starting.
I20260429 08:00:00.050030 14610 tablet_bootstrap.cc:654] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.050602 14611 tablet_bootstrap.cc:654] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.051074 14612 tablet_bootstrap.cc:654] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.051362 14610 log.cc:826] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:00.051697 14611 log.cc:826] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:00.052139 14612 log.cc:826] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:00.053334 14610 tablet_bootstrap.cc:492] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: No bootstrap required, opened a new log
I20260429 08:00:00.053563 14610 ts_tablet_manager.cc:1403] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260429 08:00:00.053889 14611 tablet_bootstrap.cc:492] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: No bootstrap required, opened a new log
I20260429 08:00:00.053921 14612 tablet_bootstrap.cc:492] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: No bootstrap required, opened a new log
I20260429 08:00:00.054045 14612 ts_tablet_manager.cc:1403] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260429 08:00:00.054046 14611 ts_tablet_manager.cc:1403] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260429 08:00:00.056129 14610 raft_consensus.cc:359] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.056279 14610 raft_consensus.cc:385] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.056320 14610 raft_consensus.cc:740] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.056672 14610 consensus_queue.cc:260] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.056881 14612 raft_consensus.cc:359] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.056881 14611 raft_consensus.cc:359] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.057060 14612 raft_consensus.cc:385] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.057060 14611 raft_consensus.cc:385] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.057106 14612 raft_consensus.cc:740] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3aaf4d86b744449e1df19a35e8722e, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.057106 14611 raft_consensus.cc:740] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.057265 14445 heartbeater.cc:499] Master 127.12.12.190:41491 was elected leader, sending a full tablet report...
I20260429 08:00:00.057528 14610 ts_tablet_manager.cc:1434] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260429 08:00:00.057511 14611 consensus_queue.cc:260] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.057511 14612 consensus_queue.cc:260] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.058302 14611 ts_tablet_manager.cc:1434] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260429 08:00:00.058302 14612 ts_tablet_manager.cc:1434] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260429 08:00:00.058634 14303 heartbeater.cc:499] Master 127.12.12.190:41491 was elected leader, sending a full tablet report...
I20260429 08:00:00.059486 14587 heartbeater.cc:499] Master 127.12.12.190:41491 was elected leader, sending a full tablet report...
W20260429 08:00:00.066838 14304 tablet.cc:2404] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 08:00:00.237044 14588 tablet.cc:2404] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 08:00:00.280555 14446 tablet.cc:2404] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:00.326092 14617 raft_consensus.cc:493] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:00.326313 14617 raft_consensus.cc:515] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.327545 14617 leader_election.cc:290] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817), 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961)
I20260429 08:00:00.330574 14620 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.327834 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.12.12.129:46701)
0429 08:00:00.328144 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.328157 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.328253 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.328543 (+ 290us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.328545 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.328563 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.328794 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.328824 (+ 30us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.329510 (+ 686us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.329516 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.330128 (+ 612us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.330135 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.330266 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.330283 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.330335 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.330387 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":62,"threads_started":1}
I20260429 08:00:00.330577 14619 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.327742 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:44817 (local address 127.12.12.129:43707)
0429 08:00:00.328007 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.328021 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.328188 (+ 167us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.328464 (+ 276us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.328467 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.328484 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.328699 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.328705 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.329448 (+ 743us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.329454 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.330048 (+ 594us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.330055 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.330198 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.330217 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.330273 (+ 56us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.330351 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260429 08:00:00.330965 14605 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.327881 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:43707 (local address 127.12.12.131:44817)
0429 08:00:00.328010 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.328017 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.328095 (+ 78us) server_negotiation.cc:408] Connection header received
0429 08:00:00.328318 (+ 223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.328321 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.328366 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.328452 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.328838 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.329337 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.330177 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.330699 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.330737 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.330788 (+ 51us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.330833 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 08:00:00.331079 14609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.328031 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:46701 (local address 127.12.12.130:45961)
0429 08:00:00.328209 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.328214 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.328229 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:00:00.328395 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.328398 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.328439 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.328529 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.328941 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.329395 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.330257 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.330773 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.330813 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.330867 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.330923 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260429 08:00:00.331673 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f0a7f20b3d4453b3db0d45317f8c48" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d0ed40de3420ea5950c5ec34fada8" is_pre_election: true
I20260429 08:00:00.331681 14541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f0a7f20b3d4453b3db0d45317f8c48" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e" is_pre_election: true
I20260429 08:00:00.331960 14399 raft_consensus.cc:2468] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 0.
I20260429 08:00:00.331970 14541 raft_consensus.cc:2468] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 0.
I20260429 08:00:00.332454 14191 leader_election.cc:304] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d0ed40de3420ea5950c5ec34fada8, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:00.332726 14617 raft_consensus.cc:2804] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:00.332818 14617 raft_consensus.cc:493] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:00.332887 14617 raft_consensus.cc:3060] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.333827 14617 raft_consensus.cc:515] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.334232 14617 leader_election.cc:290] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 election: Requested vote from peers 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817), 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961)
I20260429 08:00:00.334528 14541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f0a7f20b3d4453b3db0d45317f8c48" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e"
I20260429 08:00:00.334604 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f0a7f20b3d4453b3db0d45317f8c48" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d0ed40de3420ea5950c5ec34fada8"
I20260429 08:00:00.334667 14541 raft_consensus.cc:3060] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.334748 14399 raft_consensus.cc:3060] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.335614 14541 raft_consensus.cc:2468] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 1.
I20260429 08:00:00.335687 14399 raft_consensus.cc:2468] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 1.
I20260429 08:00:00.335968 14189 leader_election.cc:304] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3aaf4d86b744449e1df19a35e8722e, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:00.336215 14617 raft_consensus.cc:2804] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:00.336419 14617 raft_consensus.cc:697] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 1 LEADER]: Becoming Leader. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Running, Role: LEADER
I20260429 08:00:00.336721 14617 consensus_queue.cc:237] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } }
I20260429 08:00:00.339807 14096 catalog_manager.cc:5671] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129). New cstate: current_term: 1 leader_uuid: "9053f40d69cc4260a8218eaa12684cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } health_report { overall_health: UNKNOWN } } }
Apr 29 08:00:00 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:00.426219 14147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.418773 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.12.12.190:41491)
0429 08:00:00.418905 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.418909 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.419105 (+ 196us) server_negotiation.cc:408] Connection header received
0429 08:00:00.419236 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.419238 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.419280 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.419360 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:00.420148 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.420623 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.421266 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.421394 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.423399 (+ 2005us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:00.423414 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:00.423417 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:00.423438 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:00.424669 (+ 1231us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:00.425141 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:00.425144 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:00.425145 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:00.425182 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:00.425427 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:00.425428 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:00.425429 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:00.425577 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:00.425649 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.425932 (+ 283us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.426072 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":49}
I20260429 08:00:00.426221 14628 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.418738 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41491 (local address 127.0.0.1:54108)
0429 08:00:00.419010 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.419028 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.419157 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.419399 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.419402 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.419824 (+ 422us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:00.420029 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.420035 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.420717 (+ 682us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.420719 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.421174 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.421181 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.421269 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.421854 (+ 585us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:00.421867 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:00.423279 (+ 1412us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:00.424761 (+ 1482us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:00.424768 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:00.424771 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:00.425038 (+ 267us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:00.425289 (+ 251us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:00.425292 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:00.425293 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:00.425347 (+ 54us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:00.425676 (+ 329us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:00.425681 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:00.425786 (+ 105us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.425879 (+ 93us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.425998 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260429 08:00:00.430394 14096 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:54108:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 08:00:00.431124 14096 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:00.435915 14379 tablet_service.cc:1511] Processing CreateTablet for tablet 00a34b4b3493433f9215391cc892c9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0a22b03d2894f0e97d33fa503ed1d0d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 08:00:00.435779 14237 tablet_service.cc:1511] Processing CreateTablet for tablet 00a34b4b3493433f9215391cc892c9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0a22b03d2894f0e97d33fa503ed1d0d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 08:00:00.436445 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a34b4b3493433f9215391cc892c9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.436446 14379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a34b4b3493433f9215391cc892c9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.436180 14521 tablet_service.cc:1511] Processing CreateTablet for tablet 00a34b4b3493433f9215391cc892c9ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0a22b03d2894f0e97d33fa503ed1d0d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 08:00:00.436774 14521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00a34b4b3493433f9215391cc892c9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.438982 14612 tablet_bootstrap.cc:492] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: Bootstrap starting.
I20260429 08:00:00.439144 14611 tablet_bootstrap.cc:492] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: Bootstrap starting.
I20260429 08:00:00.439357 14610 tablet_bootstrap.cc:492] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: Bootstrap starting.
I20260429 08:00:00.440037 14612 tablet_bootstrap.cc:654] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.440053 14611 tablet_bootstrap.cc:654] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.440407 14610 tablet_bootstrap.cc:654] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.441391 14611 tablet_bootstrap.cc:492] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: No bootstrap required, opened a new log
I20260429 08:00:00.441480 14611 ts_tablet_manager.cc:1403] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260429 08:00:00.441565 14612 tablet_bootstrap.cc:492] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: No bootstrap required, opened a new log
I20260429 08:00:00.441638 14612 ts_tablet_manager.cc:1403] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260429 08:00:00.442001 14610 tablet_bootstrap.cc:492] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: No bootstrap required, opened a new log
I20260429 08:00:00.441972 14611 raft_consensus.cc:359] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.442076 14611 raft_consensus.cc:385] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.442087 14610 ts_tablet_manager.cc:1403] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 08:00:00.442122 14611 raft_consensus.cc:740] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.442257 14611 consensus_queue.cc:260] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.442324 14612 raft_consensus.cc:359] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.442512 14611 ts_tablet_manager.cc:1434] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 08:00:00.442553 14610 raft_consensus.cc:359] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.442649 14610 raft_consensus.cc:385] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.442520 14617 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.442679 14610 raft_consensus.cc:740] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.442734 14617 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.442798 14610 consensus_queue.cc:260] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.442441 14612 raft_consensus.cc:385] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.442907 14612 raft_consensus.cc:740] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3aaf4d86b744449e1df19a35e8722e, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.443086 14612 consensus_queue.cc:260] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.443096 14616 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.443205 14616 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.443324 14610 ts_tablet_manager.cc:1434] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:00.443379 14618 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.443488 14618 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.444710 14612 ts_tablet_manager.cc:1434] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260429 08:00:00.513206 14616 raft_consensus.cc:493] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:00.513355 14616 raft_consensus.cc:515] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.514477 14616 leader_election.cc:290] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287), 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 08:00:00.518039 14633 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.514659 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:45287 (local address 127.12.12.130:49201)
0429 08:00:00.515172 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.515188 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.515333 (+ 145us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.515637 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.515640 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.515667 (+ 27us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.516001 (+ 334us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.516009 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.516941 (+ 932us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.516946 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.517606 (+ 660us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.517614 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.517746 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.517765 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.517819 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.517873 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"spinlock_wait_cycles":27904,"thread_start_us":105,"threads_started":1}
I20260429 08:00:00.518464 14608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.514755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:49201 (local address 127.12.12.129:45287)
0429 08:00:00.514907 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.514913 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.515289 (+ 376us) server_negotiation.cc:408] Connection header received
0429 08:00:00.515458 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.515462 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.515519 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.515608 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.516128 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.516776 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.517729 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.518233 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.518271 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.518329 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.518376 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 08:00:00.518565 14632 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.514754 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:44817 (local address 127.12.12.130:35883)
0429 08:00:00.515187 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.515200 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.515333 (+ 133us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.515637 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.515640 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.515665 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:00.516002 (+ 337us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.516009 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.516964 (+ 955us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.516968 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.517751 (+ 783us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.517760 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.517890 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.517905 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.517956 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.518005 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"spinlock_wait_cycles":18560,"thread_start_us":63,"threads_started":1}
I20260429 08:00:00.518642 14605 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.514910 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:35883 (local address 127.12.12.131:44817)
0429 08:00:00.515036 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.515039 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.515288 (+ 249us) server_negotiation.cc:408] Connection header received
0429 08:00:00.515458 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.515462 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.515519 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.515608 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:00.516159 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.516806 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.517880 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.518329 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.518370 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.518431 (+ 61us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.518488 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 08:00:00.519236 14541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a34b4b3493433f9215391cc892c9ea" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e" is_pre_election: true
I20260429 08:00:00.519395 14541 raft_consensus.cc:2468] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 0.
I20260429 08:00:00.519629 14255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a34b4b3493433f9215391cc892c9ea" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9053f40d69cc4260a8218eaa12684cbf" is_pre_election: true
I20260429 08:00:00.519749 14331 leader_election.cc:304] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d0ed40de3420ea5950c5ec34fada8, 1e3aaf4d86b744449e1df19a35e8722e; no voters:
I20260429 08:00:00.519908 14255 raft_consensus.cc:2468] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 0.
I20260429 08:00:00.520113 14616 raft_consensus.cc:2804] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:00.520187 14616 raft_consensus.cc:493] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:00.520236 14616 raft_consensus.cc:3060] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.521311 14616 raft_consensus.cc:515] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.521684 14616 leader_election.cc:290] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 election: Requested vote from peers 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287), 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 08:00:00.522008 14541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a34b4b3493433f9215391cc892c9ea" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e"
I20260429 08:00:00.522008 14255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00a34b4b3493433f9215391cc892c9ea" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9053f40d69cc4260a8218eaa12684cbf"
I20260429 08:00:00.522125 14255 raft_consensus.cc:3060] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.522130 14541 raft_consensus.cc:3060] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.523175 14541 raft_consensus.cc:2468] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 1.
I20260429 08:00:00.523175 14255 raft_consensus.cc:2468] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 1.
I20260429 08:00:00.523481 14333 leader_election.cc:304] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d0ed40de3420ea5950c5ec34fada8, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:00.523660 14616 raft_consensus.cc:2804] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:00.523854 14616 raft_consensus.cc:697] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [term 1 LEADER]: Becoming Leader. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Running, Role: LEADER
I20260429 08:00:00.524204 14616 consensus_queue.cc:237] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.526353 14634 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: TxnStatusTablet state changed. Reason: New leader 121d0ed40de3420ea5950c5ec34fada8. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.526531 14634 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: This TxnStatusTablet replica's current role is: LEADER
I20260429 08:00:00.526919 14636 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 08:00:00.527133 14636 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 08:00:00.527480 14097 catalog_manager.cc:5671] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130). New cstate: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:00.546499 14628 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.543011 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.0.0.1:55986)
0429 08:00:00.543176 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.543192 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.543340 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.543673 (+ 333us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.543676 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.543683 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:00.543859 (+ 176us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.543863 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.544947 (+ 1084us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.544951 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.545365 (+ 414us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.545371 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.545499 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.545542 (+ 43us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.546171 (+ 629us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.546184 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.546268 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.546347 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260429 08:00:00.546499 14609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.543139 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.12.12.130:45961)
0429 08:00:00.543308 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.543312 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.543328 (+ 16us) server_negotiation.cc:408] Connection header received
0429 08:00:00.543481 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.543485 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.543555 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.543653 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:00.543974 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.544743 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.545481 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.545668 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.545714 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.546055 (+ 341us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.546176 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.546290 (+ 114us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.546337 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 08:00:00.551775 14255 raft_consensus.cc:1275] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Refusing update from remote peer 121d0ed40de3420ea5950c5ec34fada8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:00.551833 14541 raft_consensus.cc:1275] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Refusing update from remote peer 121d0ed40de3420ea5950c5ec34fada8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:00.552414 14634 consensus_queue.cc:1048] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:00.552579 14616 consensus_queue.cc:1048] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:00.556217 14618 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: TxnStatusTablet state changed. Reason: New leader 121d0ed40de3420ea5950c5ec34fada8. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.556336 14618 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.556463 14617 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: TxnStatusTablet state changed. Reason: New leader 121d0ed40de3420ea5950c5ec34fada8. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.556568 14617 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.558247 14634 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.558358 14634 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: This TxnStatusTablet replica's current role is: LEADER
I20260429 08:00:00.558820 14617 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.558938 14617 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 9053f40d69cc4260a8218eaa12684cbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.559176 14639 mvcc.cc:204] Tried to move back new op lower bound from 7280433563853479936 to 7280433563749916672. Current Snapshot: MvccSnapshot[applied={T|T < 7280433563853479936}]
I20260429 08:00:00.560674 14641 mvcc.cc:204] Tried to move back new op lower bound from 7280433563853479936 to 7280433563749916672. Current Snapshot: MvccSnapshot[applied={T|T < 7280433563853479936}]
I20260429 08:00:00.563895 14616 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.564023 14616 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 121d0ed40de3420ea5950c5ec34fada8: This TxnStatusTablet replica's current role is: LEADER
I20260429 08:00:00.567936 14618 tablet_replica.cc:442] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } } }
I20260429 08:00:00.568060 14618 tablet_replica.cc:445] T 00a34b4b3493433f9215391cc892c9ea P 1e3aaf4d86b744449e1df19a35e8722e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:00.573554 14608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.567614 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.12.12.129:45287)
0429 08:00:00.567750 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.567755 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.570632 (+ 2877us) server_negotiation.cc:408] Connection header received
0429 08:00:00.570821 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.570823 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.570878 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.570981 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:00.571356 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.571809 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.572485 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.572665 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.572712 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.573133 (+ 421us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.573245 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.573339 (+ 94us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.573386 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:00.583685 14609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.580567 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:49613 (local address 127.12.12.130:45961)
0429 08:00:00.580698 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.580703 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.581009 (+ 306us) server_negotiation.cc:408] Connection header received
0429 08:00:00.581178 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.581182 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.581236 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.581341 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:00.581799 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.582294 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.582976 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.583187 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.583226 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.583295 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.583384 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.583510 (+ 126us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.583549 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:00:00.583701 14651 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.580489 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.12.12.129:49613)
0429 08:00:00.580923 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.580937 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.581053 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.581439 (+ 386us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.581442 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.581449 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:00.581669 (+ 220us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.581674 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.582430 (+ 756us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.582432 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.582848 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.582854 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.582992 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.583032 (+ 40us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.583402 (+ 370us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.583408 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.583488 (+ 80us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.583550 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260429 08:00:00.590540 14541 raft_consensus.cc:1275] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Refusing update from remote peer 9053f40d69cc4260a8218eaa12684cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:00.590751 14399 raft_consensus.cc:1275] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Refusing update from remote peer 9053f40d69cc4260a8218eaa12684cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:00.591315 14617 consensus_queue.cc:1048] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:00.591493 14621 consensus_queue.cc:1048] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:00.595732 14640 mvcc.cc:204] Tried to move back new op lower bound from 7280433564014579712 to 7280433562981896192. Current Snapshot: MvccSnapshot[applied={T|T < 7280433564014579712}]
I20260429 08:00:00.612766 14608 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.609585 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:49155 (local address 127.12.12.129:45287)
0429 08:00:00.609725 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.609729 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.609930 (+ 201us) server_negotiation.cc:408] Connection header received
0429 08:00:00.610050 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.610052 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.610095 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.610161 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:00.610631 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.611343 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.612212 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.612340 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.612387 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.612451 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.612506 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.612626 (+ 120us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.612663 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 08:00:00.612843 14658 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.609514 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:45287 (local address 127.12.12.130:49155)
0429 08:00:00.609840 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.609856 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.609963 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.610193 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.610197 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.610205 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:00.610517 (+ 312us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.610523 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.611462 (+ 939us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.611466 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.612093 (+ 627us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.612101 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.612220 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.612261 (+ 41us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.612536 (+ 275us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.612542 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.612608 (+ 66us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.612664 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":125,"threads_started":1}
I20260429 08:00:00.641777 14097 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:00.641978 14097 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:00.644907 14097 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Sending DeleteTablet for 3 replicas of tablet 16f0a7f20b3d4453b3db0d45317f8c48
I20260429 08:00:00.645766 14521 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f0a7f20b3d4453b3db0d45317f8c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:00 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:41564
I20260429 08:00:00.646088 14237 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f0a7f20b3d4453b3db0d45317f8c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:00 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:44160
I20260429 08:00:00.646513 14379 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f0a7f20b3d4453b3db0d45317f8c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:00 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:55976
I20260429 08:00:00.646723 14662 tablet_replica.cc:333] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: stopping tablet replica
I20260429 08:00:00.647058 14662 raft_consensus.cc:2243] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:00.647563 14662 raft_consensus.cc:2272] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:00.647790 14097 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:00.648243 14097 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:00.649215 14662 ts_tablet_manager.cc:1916] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:00.649755 14661 tablet_replica.cc:333] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: stopping tablet replica
I20260429 08:00:00.650075 14663 tablet_replica.cc:333] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: stopping tablet replica
I20260429 08:00:00.650408 14663 raft_consensus.cc:2243] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:00.650720 14661 raft_consensus.cc:2243] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:00.650991 14661 raft_consensus.cc:2272] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:00.651217 14663 raft_consensus.cc:2272] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:00.652835 14663 ts_tablet_manager.cc:1916] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:00.654054 14661 ts_tablet_manager.cc:1916] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:00.655038 14521 tablet_service.cc:1511] Processing CreateTablet for tablet e769a172a0fa461aae1040a604477560 (DEFAULT_TABLE table=test-table [id=0e23703a5bad4a4fae9c96d3a0f9ff64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:00.655305 14521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e769a172a0fa461aae1040a604477560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.657691 14612 tablet_bootstrap.cc:492] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Bootstrap starting.
I20260429 08:00:00.657796 14661 ts_tablet_manager.cc:1929] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:00.657873 14661 log.cc:1199] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/16f0a7f20b3d4453b3db0d45317f8c48
I20260429 08:00:00.658156 14661 ts_tablet_manager.cc:1950] T 16f0a7f20b3d4453b3db0d45317f8c48 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting consensus metadata
I20260429 08:00:00.658681 14612 tablet_bootstrap.cc:654] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.659740 14083 catalog_manager.cc:5002] TS 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817): tablet 16f0a7f20b3d4453b3db0d45317f8c48 (table test-table [id=bcf242d6c5e640a8ba40bfa6874d2ba8]) successfully deleted
I20260429 08:00:00.660843 14379 tablet_service.cc:1511] Processing CreateTablet for tablet e769a172a0fa461aae1040a604477560 (DEFAULT_TABLE table=test-table [id=0e23703a5bad4a4fae9c96d3a0f9ff64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:00.660985 14237 tablet_service.cc:1511] Processing CreateTablet for tablet e769a172a0fa461aae1040a604477560 (DEFAULT_TABLE table=test-table [id=0e23703a5bad4a4fae9c96d3a0f9ff64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:00.661125 14379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e769a172a0fa461aae1040a604477560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.661928 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e769a172a0fa461aae1040a604477560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:00.663681 14610 tablet_bootstrap.cc:492] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Bootstrap starting.
I20260429 08:00:00.664219 14611 tablet_bootstrap.cc:492] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Bootstrap starting.
I20260429 08:00:00.664673 14610 tablet_bootstrap.cc:654] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.665169 14611 tablet_bootstrap.cc:654] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:00.665594 14612 tablet_bootstrap.cc:492] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: No bootstrap required, opened a new log
I20260429 08:00:00.665696 14612 ts_tablet_manager.cc:1403] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260429 08:00:00.666224 14612 raft_consensus.cc:359] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.666337 14612 raft_consensus.cc:385] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.666371 14612 raft_consensus.cc:740] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3aaf4d86b744449e1df19a35e8722e, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.666488 14612 consensus_queue.cc:260] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.666528 14663 ts_tablet_manager.cc:1929] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:00.666615 14663 log.cc:1199] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/16f0a7f20b3d4453b3db0d45317f8c48
I20260429 08:00:00.666767 14612 ts_tablet_manager.cc:1434] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 08:00:00.666957 14663 ts_tablet_manager.cc:1950] T 16f0a7f20b3d4453b3db0d45317f8c48 P 121d0ed40de3420ea5950c5ec34fada8: Deleting consensus metadata
I20260429 08:00:00.668120 14085 catalog_manager.cc:5002] TS 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961): tablet 16f0a7f20b3d4453b3db0d45317f8c48 (table test-table [id=bcf242d6c5e640a8ba40bfa6874d2ba8]) successfully deleted
I20260429 08:00:00.669049 14610 tablet_bootstrap.cc:492] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: No bootstrap required, opened a new log
I20260429 08:00:00.669142 14610 ts_tablet_manager.cc:1403] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260429 08:00:00.669723 14610 raft_consensus.cc:359] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.669824 14610 raft_consensus.cc:385] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.669855 14610 raft_consensus.cc:740] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.669967 14610 consensus_queue.cc:260] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.670207 14610 ts_tablet_manager.cc:1434] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:00.673593 14611 tablet_bootstrap.cc:492] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: No bootstrap required, opened a new log
I20260429 08:00:00.673691 14611 ts_tablet_manager.cc:1403] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260429 08:00:00.674157 14611 raft_consensus.cc:359] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.674257 14611 raft_consensus.cc:385] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:00.674288 14611 raft_consensus.cc:740] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Initialized, Role: FOLLOWER
I20260429 08:00:00.674415 14611 consensus_queue.cc:260] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.674649 14611 ts_tablet_manager.cc:1434] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 08:00:00.675096 14662 ts_tablet_manager.cc:1929] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:00.675223 14662 log.cc:1199] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/16f0a7f20b3d4453b3db0d45317f8c48
I20260429 08:00:00.675583 14662 ts_tablet_manager.cc:1950] T 16f0a7f20b3d4453b3db0d45317f8c48 P 9053f40d69cc4260a8218eaa12684cbf: Deleting consensus metadata
I20260429 08:00:00.676661 14085 catalog_manager.cc:5002] TS 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287): tablet 16f0a7f20b3d4453b3db0d45317f8c48 (table test-table [id=bcf242d6c5e640a8ba40bfa6874d2ba8]) successfully deleted
I20260429 08:00:00.706086 14634 raft_consensus.cc:493] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:00.706247 14634 raft_consensus.cc:515] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.706784 14634 leader_election.cc:290] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287), 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 08:00:00.707298 14255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e769a172a0fa461aae1040a604477560" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9053f40d69cc4260a8218eaa12684cbf" is_pre_election: true
I20260429 08:00:00.707470 14255 raft_consensus.cc:2468] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 0.
I20260429 08:00:00.707922 14541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e769a172a0fa461aae1040a604477560" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e" is_pre_election: true
I20260429 08:00:00.708074 14541 raft_consensus.cc:2468] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 0.
I20260429 08:00:00.708177 14333 leader_election.cc:304] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d0ed40de3420ea5950c5ec34fada8, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:00.708389 14634 raft_consensus.cc:2804] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:00.708448 14634 raft_consensus.cc:493] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:00.708485 14634 raft_consensus.cc:3060] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.709712 14634 raft_consensus.cc:515] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.710189 14634 leader_election.cc:290] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 election: Requested vote from peers 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287), 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 08:00:00.710570 14255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e769a172a0fa461aae1040a604477560" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9053f40d69cc4260a8218eaa12684cbf"
I20260429 08:00:00.710667 14541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e769a172a0fa461aae1040a604477560" candidate_uuid: "121d0ed40de3420ea5950c5ec34fada8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e"
I20260429 08:00:00.710705 14255 raft_consensus.cc:3060] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.710777 14541 raft_consensus.cc:3060] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:00.711853 14541 raft_consensus.cc:2468] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 1.
I20260429 08:00:00.711910 14255 raft_consensus.cc:2468] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d0ed40de3420ea5950c5ec34fada8 in term 1.
I20260429 08:00:00.712203 14331 leader_election.cc:304] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d0ed40de3420ea5950c5ec34fada8, 1e3aaf4d86b744449e1df19a35e8722e; no voters:
I20260429 08:00:00.712401 14634 raft_consensus.cc:2804] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:00.712510 14634 raft_consensus.cc:697] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 LEADER]: Becoming Leader. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Running, Role: LEADER
I20260429 08:00:00.712641 14634 consensus_queue.cc:237] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:00.714481 14097 catalog_manager.cc:5671] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130). New cstate: current_term: 1 leader_uuid: "121d0ed40de3420ea5950c5ec34fada8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:00.731810 14609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.728600 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.12.12.130:45961)
0429 08:00:00.728717 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.728722 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.728785 (+ 63us) server_negotiation.cc:408] Connection header received
0429 08:00:00.728944 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.728948 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.729007 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.729088 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:00.729509 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.730197 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.731081 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.731276 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.731341 (+ 65us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.731433 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.731508 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.731626 (+ 118us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.731667 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 08:00:00.741783 14609 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.738753 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:36953 (local address 127.12.12.130:45961)
0429 08:00:00.738850 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:00.738854 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:00.738865 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:00.738901 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:00.738905 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:00.738949 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:00.739038 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:00.739406 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.740150 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.741020 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.741210 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.741245 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.741330 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.741444 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:00.741560 (+ 116us) server_negotiation.cc:300] Negotiation successful
0429 08:00:00.741601 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260429 08:00:00.741791 14658 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:00.738578 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.12.12.130:36953)
0429 08:00:00.738720 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:00.738736 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:00.738820 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:00.739047 (+ 227us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:00.739050 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:00.739059 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:00.739304 (+ 245us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.739311 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.740256 (+ 945us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:00.740260 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:00.740912 (+ 652us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:00.740922 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:00.741035 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:00.741072 (+ 37us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:00.741462 (+ 390us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:00.741468 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 08:00:00.741537 (+ 69us) client_negotiation.cc:241] Negotiation successful
0429 08:00:00.741606 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260429 08:00:00.749430 14541 raft_consensus.cc:1275] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Refusing update from remote peer 121d0ed40de3420ea5950c5ec34fada8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:00.749549 14255 raft_consensus.cc:1275] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Refusing update from remote peer 121d0ed40de3420ea5950c5ec34fada8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:00.750007 14652 consensus_queue.cc:1048] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:00.750195 14634 consensus_queue.cc:1048] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260429 08:00:00.765688 14656 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260429 08:00:02.331177 14098 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:02.331342 14098 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:02.333794 14098 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Sending DeleteTablet for 3 replicas of tablet e769a172a0fa461aae1040a604477560
I20260429 08:00:02.334386 14237 tablet_service.cc:1558] Processing DeleteTablet for tablet e769a172a0fa461aae1040a604477560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:02 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:44160
I20260429 08:00:02.334486 14378 tablet_service.cc:1558] Processing DeleteTablet for tablet e769a172a0fa461aae1040a604477560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:02 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:55976
I20260429 08:00:02.334597 14521 tablet_service.cc:1558] Processing DeleteTablet for tablet e769a172a0fa461aae1040a604477560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:02 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:41564
I20260429 08:00:02.334733 14691 tablet_replica.cc:333] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: stopping tablet replica
I20260429 08:00:02.334832 14692 tablet_replica.cc:333] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: stopping tablet replica
I20260429 08:00:02.334889 14691 raft_consensus.cc:2243] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:02.334939 14693 tablet_replica.cc:333] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: stopping tablet replica
I20260429 08:00:02.335011 14692 raft_consensus.cc:2243] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:02.335059 14693 raft_consensus.cc:2243] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:02.335075 14691 raft_consensus.cc:2272] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:02.335409 14692 raft_consensus.cc:2272] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:02.335464 14693 raft_consensus.cc:2272] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:02.335932 14691 ts_tablet_manager.cc:1916] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:02.336097 14693 ts_tablet_manager.cc:1916] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:02.336095 14692 ts_tablet_manager.cc:1916] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:02.338438 14693 ts_tablet_manager.cc:1929] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 08:00:02.338585 14693 log.cc:1199] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/e769a172a0fa461aae1040a604477560
I20260429 08:00:02.338591 14691 ts_tablet_manager.cc:1929] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 08:00:02.338649 14691 log.cc:1199] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/e769a172a0fa461aae1040a604477560
I20260429 08:00:02.338964 14693 ts_tablet_manager.cc:1950] T e769a172a0fa461aae1040a604477560 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting consensus metadata
I20260429 08:00:02.338966 14691 ts_tablet_manager.cc:1950] T e769a172a0fa461aae1040a604477560 P 9053f40d69cc4260a8218eaa12684cbf: Deleting consensus metadata
I20260429 08:00:02.339052 14692 ts_tablet_manager.cc:1929] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 08:00:02.339180 14692 log.cc:1199] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/e769a172a0fa461aae1040a604477560
I20260429 08:00:02.339560 14692 ts_tablet_manager.cc:1950] T e769a172a0fa461aae1040a604477560 P 121d0ed40de3420ea5950c5ec34fada8: Deleting consensus metadata
I20260429 08:00:02.339673 14085 catalog_manager.cc:5002] TS 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287): tablet e769a172a0fa461aae1040a604477560 (table test-table [id=0e23703a5bad4a4fae9c96d3a0f9ff64]) successfully deleted
I20260429 08:00:02.339948 14083 catalog_manager.cc:5002] TS 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817): tablet e769a172a0fa461aae1040a604477560 (table test-table [id=0e23703a5bad4a4fae9c96d3a0f9ff64]) successfully deleted
I20260429 08:00:02.340296 14085 catalog_manager.cc:5002] TS 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961): tablet e769a172a0fa461aae1040a604477560 (table test-table [id=0e23703a5bad4a4fae9c96d3a0f9ff64]) successfully deleted
Apr 29 08:00:02 dist-test-slave-nhqz krb5kdc[14050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 08:00:02 dist-test-slave-nhqz krb5kdc[14050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:02.360651 14701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.352857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.12.12.190:41491)
0429 08:00:02.353154 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.353159 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.353173 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:00:02.353313 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.353316 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.353396 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.353494 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:02.354267 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.354874 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.355522 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.355735 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.357756 (+ 2021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:02.357779 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:02.357782 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:02.357806 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:02.359149 (+ 1343us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:02.359528 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:02.359532 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:02.359536 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:02.359582 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:02.359810 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:02.359813 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:02.359815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:02.359975 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:02.360130 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.360347 (+ 217us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.360475 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260429 08:00:02.363174 14098 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:02.363636 14098 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:02.368472 14237 tablet_service.cc:1511] Processing CreateTablet for tablet 710c131dab2c48ce8e6a6bd9414cdcc5 (DEFAULT_TABLE table=test-table [id=05f0b773d6814c84ac3dcf00927f8b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:02.368906 14521 tablet_service.cc:1511] Processing CreateTablet for tablet 710c131dab2c48ce8e6a6bd9414cdcc5 (DEFAULT_TABLE table=test-table [id=05f0b773d6814c84ac3dcf00927f8b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:02.369199 14521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710c131dab2c48ce8e6a6bd9414cdcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:02.369257 14378 tablet_service.cc:1511] Processing CreateTablet for tablet 710c131dab2c48ce8e6a6bd9414cdcc5 (DEFAULT_TABLE table=test-table [id=05f0b773d6814c84ac3dcf00927f8b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:02.369470 14378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710c131dab2c48ce8e6a6bd9414cdcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:02.370997 14702 tablet_bootstrap.cc:492] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Bootstrap starting.
I20260429 08:00:02.368803 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710c131dab2c48ce8e6a6bd9414cdcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:02.371747 14703 tablet_bootstrap.cc:492] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Bootstrap starting.
I20260429 08:00:02.371949 14702 tablet_bootstrap.cc:654] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:02.372470 14703 tablet_bootstrap.cc:654] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:02.373201 14702 tablet_bootstrap.cc:492] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: No bootstrap required, opened a new log
I20260429 08:00:02.373292 14702 ts_tablet_manager.cc:1403] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:02.373751 14703 tablet_bootstrap.cc:492] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: No bootstrap required, opened a new log
I20260429 08:00:02.373826 14703 ts_tablet_manager.cc:1403] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 08:00:02.373816 14702 raft_consensus.cc:359] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.373924 14702 raft_consensus.cc:385] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:02.373993 14702 raft_consensus.cc:740] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3aaf4d86b744449e1df19a35e8722e, State: Initialized, Role: FOLLOWER
I20260429 08:00:02.374224 14702 consensus_queue.cc:260] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.374240 14703 raft_consensus.cc:359] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.374327 14703 raft_consensus.cc:385] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:02.374393 14703 raft_consensus.cc:740] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Initialized, Role: FOLLOWER
I20260429 08:00:02.374521 14703 consensus_queue.cc:260] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.374694 14702 ts_tablet_manager.cc:1434] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 08:00:02.374809 14703 ts_tablet_manager.cc:1434] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 08:00:02.374950 14706 tablet_bootstrap.cc:492] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Bootstrap starting.
I20260429 08:00:02.375931 14706 tablet_bootstrap.cc:654] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:02.377154 14706 tablet_bootstrap.cc:492] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: No bootstrap required, opened a new log
I20260429 08:00:02.377312 14706 ts_tablet_manager.cc:1403] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:02.377820 14706 raft_consensus.cc:359] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.377945 14706 raft_consensus.cc:385] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:02.378006 14706 raft_consensus.cc:740] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Initialized, Role: FOLLOWER
I20260429 08:00:02.378121 14706 consensus_queue.cc:260] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.378539 14706 ts_tablet_manager.cc:1434] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 08:00:02.569742 14709 raft_consensus.cc:493] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:02.569871 14709 raft_consensus.cc:515] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.570416 14709 leader_election.cc:290] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961), 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 08:00:02.570894 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "710c131dab2c48ce8e6a6bd9414cdcc5" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d0ed40de3420ea5950c5ec34fada8" is_pre_election: true
I20260429 08:00:02.570917 14539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "710c131dab2c48ce8e6a6bd9414cdcc5" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e" is_pre_election: true
I20260429 08:00:02.571051 14399 raft_consensus.cc:2468] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 0.
I20260429 08:00:02.571065 14539 raft_consensus.cc:2468] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 0.
I20260429 08:00:02.571410 14189 leader_election.cc:304] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3aaf4d86b744449e1df19a35e8722e, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:02.571620 14709 raft_consensus.cc:2804] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:02.571683 14709 raft_consensus.cc:493] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:02.571746 14709 raft_consensus.cc:3060] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:02.572896 14709 raft_consensus.cc:515] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.573239 14709 leader_election.cc:290] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 election: Requested vote from peers 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961), 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817)
I20260429 08:00:02.573573 14539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "710c131dab2c48ce8e6a6bd9414cdcc5" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e3aaf4d86b744449e1df19a35e8722e"
I20260429 08:00:02.573573 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "710c131dab2c48ce8e6a6bd9414cdcc5" candidate_uuid: "9053f40d69cc4260a8218eaa12684cbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d0ed40de3420ea5950c5ec34fada8"
I20260429 08:00:02.573693 14399 raft_consensus.cc:3060] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:02.573704 14539 raft_consensus.cc:3060] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:02.574522 14539 raft_consensus.cc:2468] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 1.
I20260429 08:00:02.574689 14399 raft_consensus.cc:2468] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9053f40d69cc4260a8218eaa12684cbf in term 1.
I20260429 08:00:02.574821 14189 leader_election.cc:304] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3aaf4d86b744449e1df19a35e8722e, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:02.575004 14709 raft_consensus.cc:2804] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:02.575295 14709 raft_consensus.cc:697] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 1 LEADER]: Becoming Leader. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Running, Role: LEADER
I20260429 08:00:02.575460 14709 consensus_queue.cc:237] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.577203 14097 catalog_manager.cc:5671] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129). New cstate: current_term: 1 leader_uuid: "9053f40d69cc4260a8218eaa12684cbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:02.615212 14713 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.612551 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.12.12.129:45287)
0429 08:00:02.612856 (+ 305us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.612859 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.612869 (+ 10us) server_negotiation.cc:408] Connection header received
0429 08:00:02.612905 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.612907 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.612981 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.613070 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:02.613422 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.613929 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.614565 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.614694 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.614726 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:02.614792 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:02.614859 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.615036 (+ 177us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.615072 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260429 08:00:02.621546 14539 raft_consensus.cc:1275] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Refusing update from remote peer 9053f40d69cc4260a8218eaa12684cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:02.621588 14399 raft_consensus.cc:1275] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Refusing update from remote peer 9053f40d69cc4260a8218eaa12684cbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:02.622113 14709 consensus_queue.cc:1048] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:02.622339 14710 consensus_queue.cc:1048] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:02.655160 14097 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:02.655313 14097 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:02.657416 14097 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Sending DeleteTablet for 3 replicas of tablet 710c131dab2c48ce8e6a6bd9414cdcc5
I20260429 08:00:02.658094 14521 tablet_service.cc:1558] Processing DeleteTablet for tablet 710c131dab2c48ce8e6a6bd9414cdcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:02 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:41564
I20260429 08:00:02.658120 14378 tablet_service.cc:1558] Processing DeleteTablet for tablet 710c131dab2c48ce8e6a6bd9414cdcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:02 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:55976
I20260429 08:00:02.658242 14237 tablet_service.cc:1558] Processing DeleteTablet for tablet 710c131dab2c48ce8e6a6bd9414cdcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:02 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:44160
I20260429 08:00:02.658317 14693 tablet_replica.cc:333] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: stopping tablet replica
I20260429 08:00:02.658453 14691 tablet_replica.cc:333] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: stopping tablet replica
I20260429 08:00:02.658456 14693 raft_consensus.cc:2243] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:02.658581 14691 raft_consensus.cc:2243] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:02.658603 14693 raft_consensus.cc:2272] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:02.658818 14692 tablet_replica.cc:333] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: stopping tablet replica
I20260429 08:00:02.658922 14692 raft_consensus.cc:2243] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:02.658958 14691 raft_consensus.cc:2272] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:02.659024 14692 raft_consensus.cc:2272] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:02.659493 14693 ts_tablet_manager.cc:1916] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:02.659765 14691 ts_tablet_manager.cc:1916] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:02.659704 14097 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:02.660146 14097 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:02.661324 14692 ts_tablet_manager.cc:1916] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:02.661798 14693 ts_tablet_manager.cc:1929] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:02.661888 14693 log.cc:1199] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/710c131dab2c48ce8e6a6bd9414cdcc5
I20260429 08:00:02.662130 14693 ts_tablet_manager.cc:1950] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 1e3aaf4d86b744449e1df19a35e8722e: Deleting consensus metadata
I20260429 08:00:02.662140 14691 ts_tablet_manager.cc:1929] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:02.662206 14691 log.cc:1199] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/710c131dab2c48ce8e6a6bd9414cdcc5
I20260429 08:00:02.662468 14691 ts_tablet_manager.cc:1950] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 9053f40d69cc4260a8218eaa12684cbf: Deleting consensus metadata
I20260429 08:00:02.662990 14083 catalog_manager.cc:5002] TS 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817): tablet 710c131dab2c48ce8e6a6bd9414cdcc5 (table test-table [id=05f0b773d6814c84ac3dcf00927f8b20]) successfully deleted
I20260429 08:00:02.663152 14085 catalog_manager.cc:5002] TS 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287): tablet 710c131dab2c48ce8e6a6bd9414cdcc5 (table test-table [id=05f0b773d6814c84ac3dcf00927f8b20]) successfully deleted
I20260429 08:00:02.663717 14692 ts_tablet_manager.cc:1929] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:02.663820 14692 log.cc:1199] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/710c131dab2c48ce8e6a6bd9414cdcc5
I20260429 08:00:02.664063 14692 ts_tablet_manager.cc:1950] T 710c131dab2c48ce8e6a6bd9414cdcc5 P 121d0ed40de3420ea5950c5ec34fada8: Deleting consensus metadata
I20260429 08:00:02.664739 14085 catalog_manager.cc:5002] TS 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961): tablet 710c131dab2c48ce8e6a6bd9414cdcc5 (table test-table [id=05f0b773d6814c84ac3dcf00927f8b20]) successfully deleted
I20260429 08:00:02.666594 14237 tablet_service.cc:1511] Processing CreateTablet for tablet 93b0559276304aee9dac11f9274058dd (DEFAULT_TABLE table=test-table [id=79156c2a014f433996cbf68950a05844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:02.666736 14378 tablet_service.cc:1511] Processing CreateTablet for tablet 93b0559276304aee9dac11f9274058dd (DEFAULT_TABLE table=test-table [id=79156c2a014f433996cbf68950a05844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:02.666864 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93b0559276304aee9dac11f9274058dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:02.666947 14378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93b0559276304aee9dac11f9274058dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:02.666947 14521 tablet_service.cc:1511] Processing CreateTablet for tablet 93b0559276304aee9dac11f9274058dd (DEFAULT_TABLE table=test-table [id=79156c2a014f433996cbf68950a05844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:02.667228 14521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93b0559276304aee9dac11f9274058dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:02.668915 14702 tablet_bootstrap.cc:492] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Bootstrap starting.
I20260429 08:00:02.669059 14706 tablet_bootstrap.cc:492] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf: Bootstrap starting.
I20260429 08:00:02.669859 14706 tablet_bootstrap.cc:654] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:02.669948 14702 tablet_bootstrap.cc:654] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:02.670536 14703 tablet_bootstrap.cc:492] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Bootstrap starting.
I20260429 08:00:02.670964 14706 tablet_bootstrap.cc:492] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf: No bootstrap required, opened a new log
I20260429 08:00:02.671062 14706 ts_tablet_manager.cc:1403] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 08:00:02.671169 14702 tablet_bootstrap.cc:492] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: No bootstrap required, opened a new log
I20260429 08:00:02.671240 14702 ts_tablet_manager.cc:1403] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260429 08:00:02.671394 14703 tablet_bootstrap.cc:654] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:02.671535 14706 raft_consensus.cc:359] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.671643 14706 raft_consensus.cc:385] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:02.671679 14706 raft_consensus.cc:740] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9053f40d69cc4260a8218eaa12684cbf, State: Initialized, Role: FOLLOWER
I20260429 08:00:02.671676 14702 raft_consensus.cc:359] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.671772 14702 raft_consensus.cc:385] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:02.671802 14702 raft_consensus.cc:740] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3aaf4d86b744449e1df19a35e8722e, State: Initialized, Role: FOLLOWER
I20260429 08:00:02.671804 14706 consensus_queue.cc:260] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.671913 14702 consensus_queue.cc:260] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.672060 14706 ts_tablet_manager.cc:1434] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 08:00:02.672175 14702 ts_tablet_manager.cc:1434] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 08:00:02.672453 14703 tablet_bootstrap.cc:492] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: No bootstrap required, opened a new log
I20260429 08:00:02.672538 14703 ts_tablet_manager.cc:1403] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 08:00:02.673010 14703 raft_consensus.cc:359] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.673125 14703 raft_consensus.cc:385] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:02.673158 14703 raft_consensus.cc:740] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d0ed40de3420ea5950c5ec34fada8, State: Initialized, Role: FOLLOWER
I20260429 08:00:02.673292 14703 consensus_queue.cc:260] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.673569 14703 ts_tablet_manager.cc:1434] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:02.763548 14707 raft_consensus.cc:493] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:02.763692 14707 raft_consensus.cc:515] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.764672 14707 leader_election.cc:290] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287), 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961)
I20260429 08:00:02.768374 14721 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.764991 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:45287 (local address 127.12.12.131:48241)
0429 08:00:02.765265 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:02.765278 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:02.765445 (+ 167us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:02.765786 (+ 341us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:02.765789 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:02.765822 (+ 33us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:02.766092 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.766099 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.767010 (+ 911us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.767013 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:02.767924 (+ 911us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.767932 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.768035 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.768056 (+ 21us) client_negotiation.cc:770] Sending connection context
0429 08:00:02.768109 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 08:00:02.768173 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"spinlock_wait_cycles":3456,"thread_start_us":117,"threads_started":1}
I20260429 08:00:02.768424 14722 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.765438 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.12.12.131:53149)
0429 08:00:02.765714 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:02.765729 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:02.765822 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:02.766063 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:02.766066 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:02.766084 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:02.766333 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.766339 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.767281 (+ 942us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.767284 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:02.768092 (+ 808us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.768114 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.768202 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.768213 (+ 11us) client_negotiation.cc:770] Sending connection context
0429 08:00:02.768258 (+ 45us) client_negotiation.cc:241] Negotiation successful
0429 08:00:02.768294 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260429 08:00:02.768814 14713 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.765050 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:48241 (local address 127.12.12.129:45287)
0429 08:00:02.765210 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.765214 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.765380 (+ 166us) server_negotiation.cc:408] Connection header received
0429 08:00:02.765564 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.765567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.765625 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.765759 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:02.766214 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.766902 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.768046 (+ 1144us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.768545 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.768588 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.768645 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.768700 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260429 08:00:02.769259 14242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93b0559276304aee9dac11f9274058dd" candidate_uuid: "1e3aaf4d86b744449e1df19a35e8722e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9053f40d69cc4260a8218eaa12684cbf" is_pre_election: true
I20260429 08:00:02.769385 14723 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.765521 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:53149 (local address 127.12.12.130:45961)
0429 08:00:02.765849 (+ 328us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.765855 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.765871 (+ 16us) server_negotiation.cc:408] Connection header received
0429 08:00:02.765923 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.765926 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.765963 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.766031 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:02.766440 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.767188 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.768216 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.769108 (+ 892us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.769144 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.769196 (+ 52us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.769246 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260429 08:00:02.769425 14242 raft_consensus.cc:2468] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e3aaf4d86b744449e1df19a35e8722e in term 0.
I20260429 08:00:02.769806 14475 leader_election.cc:304] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3aaf4d86b744449e1df19a35e8722e, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:02.770069 14707 raft_consensus.cc:2804] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:02.770048 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93b0559276304aee9dac11f9274058dd" candidate_uuid: "1e3aaf4d86b744449e1df19a35e8722e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d0ed40de3420ea5950c5ec34fada8" is_pre_election: true
I20260429 08:00:02.770159 14707 raft_consensus.cc:493] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:02.770219 14707 raft_consensus.cc:3060] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:02.770216 14399 raft_consensus.cc:2468] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e3aaf4d86b744449e1df19a35e8722e in term 0.
I20260429 08:00:02.771246 14707 raft_consensus.cc:515] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.771662 14707 leader_election.cc:290] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [CANDIDATE]: Term 1 election: Requested vote from peers 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287), 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961)
I20260429 08:00:02.771984 14242 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93b0559276304aee9dac11f9274058dd" candidate_uuid: "1e3aaf4d86b744449e1df19a35e8722e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9053f40d69cc4260a8218eaa12684cbf"
I20260429 08:00:02.772002 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93b0559276304aee9dac11f9274058dd" candidate_uuid: "1e3aaf4d86b744449e1df19a35e8722e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d0ed40de3420ea5950c5ec34fada8"
I20260429 08:00:02.772099 14242 raft_consensus.cc:3060] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:02.772109 14399 raft_consensus.cc:3060] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:02.773157 14242 raft_consensus.cc:2468] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e3aaf4d86b744449e1df19a35e8722e in term 1.
I20260429 08:00:02.773186 14399 raft_consensus.cc:2468] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e3aaf4d86b744449e1df19a35e8722e in term 1.
I20260429 08:00:02.773435 14475 leader_election.cc:304] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3aaf4d86b744449e1df19a35e8722e, 9053f40d69cc4260a8218eaa12684cbf; no voters:
I20260429 08:00:02.773613 14707 raft_consensus.cc:2804] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:02.773844 14707 raft_consensus.cc:697] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 1 LEADER]: Becoming Leader. State: Replica: 1e3aaf4d86b744449e1df19a35e8722e, State: Running, Role: LEADER
I20260429 08:00:02.774116 14707 consensus_queue.cc:237] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } }
I20260429 08:00:02.776171 14095 catalog_manager.cc:5671] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131). New cstate: current_term: 1 leader_uuid: "1e3aaf4d86b744449e1df19a35e8722e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e3aaf4d86b744449e1df19a35e8722e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44817 } health_report { overall_health: HEALTHY } } }
I20260429 08:00:02.806169 14726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.802857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.12.12.131:44817)
0429 08:00:02.803194 (+ 337us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.803198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.803207 (+ 9us) server_negotiation.cc:408] Connection header received
0429 08:00:02.803276 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.803278 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.803326 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.803451 (+ 125us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:02.803797 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.804319 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.805095 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.805292 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.805346 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:02.805746 (+ 400us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:02.805834 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.805916 (+ 82us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.805956 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260429 08:00:02.815642 14723 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.812875 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:59007 (local address 127.12.12.130:45961)
0429 08:00:02.813032 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.813039 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.813053 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:00:02.813208 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.813213 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.813267 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.813354 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:02.813747 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.814237 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.814931 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.815141 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.815181 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:02.815250 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:02.815336 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.815467 (+ 131us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.815506 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 08:00:02.815891 14729 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.812638 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:45961 (local address 127.12.12.131:59007)
0429 08:00:02.812976 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:02.812992 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:02.813094 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:02.813378 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:02.813381 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:02.813388 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:02.813627 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.813632 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.814369 (+ 737us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.814379 (+ 10us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:02.814816 (+ 437us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.814822 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.814946 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.814984 (+ 38us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:02.815346 (+ 362us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:02.815365 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:00:02.815443 (+ 78us) client_negotiation.cc:241] Negotiation successful
0429 08:00:02.815507 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260429 08:00:02.821785 14242 raft_consensus.cc:1275] T 93b0559276304aee9dac11f9274058dd P 9053f40d69cc4260a8218eaa12684cbf [term 1 FOLLOWER]: Refusing update from remote peer 1e3aaf4d86b744449e1df19a35e8722e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:02.821779 14399 raft_consensus.cc:1275] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Refusing update from remote peer 1e3aaf4d86b744449e1df19a35e8722e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:02.822266 14724 consensus_queue.cc:1048] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9053f40d69cc4260a8218eaa12684cbf" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:02.822443 14707 consensus_queue.cc:1048] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [LEADER]: Connected to new peer: Peer: permanent_uuid: "121d0ed40de3420ea5950c5ec34fada8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:02.838389 14734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.835522 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:44817 (local address 127.12.12.130:48245)
0429 08:00:02.835774 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:02.835787 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:02.835868 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:02.836100 (+ 232us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:02.836109 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:02.836118 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:02.836434 (+ 316us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.836441 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.837196 (+ 755us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.837201 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:02.837596 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:02.837602 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.837726 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.837750 (+ 24us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:02.838085 (+ 335us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:02.838092 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 08:00:02.838192 (+ 100us) client_negotiation.cc:241] Negotiation successful
0429 08:00:02.838265 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260429 08:00:02.838415 14726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:02.835587 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:48245 (local address 127.12.12.131:44817)
0429 08:00:02.835714 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:02.835720 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:02.835854 (+ 134us) server_negotiation.cc:408] Connection header received
0429 08:00:02.835942 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:02.835947 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:02.836006 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:02.836081 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:02.836546 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.837090 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:02.837707 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:02.837898 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:02.837936 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:02.837997 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:02.838079 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:02.838219 (+ 140us) server_negotiation.cc:300] Negotiation successful
0429 08:00:02.838281 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260429 08:00:02.839329 14675 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260429 08:00:04.399830 14098 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:04.399998 14098 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:04.404542 14098 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab9e7df8a02348eca3566c1a02f2bae5: Sending DeleteTablet for 3 replicas of tablet 93b0559276304aee9dac11f9274058dd
I20260429 08:00:04.405220 14237 tablet_service.cc:1558] Processing DeleteTablet for tablet 93b0559276304aee9dac11f9274058dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:04 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:44160
I20260429 08:00:04.405229 14521 tablet_service.cc:1558] Processing DeleteTablet for tablet 93b0559276304aee9dac11f9274058dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:04 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:41564
I20260429 08:00:04.405520 14378 tablet_service.cc:1558] Processing DeleteTablet for tablet 93b0559276304aee9dac11f9274058dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:04 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:55976
I20260429 08:00:04.405560 14758 tablet_replica.cc:333] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: stopping tablet replica
I20260429 08:00:04.405570 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 14172
I20260429 08:00:04.405676 14758 raft_consensus.cc:2243] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:04.406384 14759 tablet_replica.cc:333] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: stopping tablet replica
I20260429 08:00:04.406456 14758 raft_consensus.cc:2272] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:04.406574 14759 raft_consensus.cc:2243] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:04.406749 14759 raft_consensus.cc:2272] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:04.407168 14758 ts_tablet_manager.cc:1916] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:04.407413 14759 ts_tablet_manager.cc:1916] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:04.409667 14758 ts_tablet_manager.cc:1929] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 08:00:04.409777 14758 log.cc:1199] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/93b0559276304aee9dac11f9274058dd
I20260429 08:00:04.409796 14759 ts_tablet_manager.cc:1929] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 08:00:04.409876 14759 log.cc:1199] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/93b0559276304aee9dac11f9274058dd
I20260429 08:00:04.410094 14758 ts_tablet_manager.cc:1950] T 93b0559276304aee9dac11f9274058dd P 1e3aaf4d86b744449e1df19a35e8722e: Deleting consensus metadata
I20260429 08:00:04.410208 14759 ts_tablet_manager.cc:1950] T 93b0559276304aee9dac11f9274058dd P 121d0ed40de3420ea5950c5ec34fada8: Deleting consensus metadata
I20260429 08:00:04.410807 14085 catalog_manager.cc:5002] TS 121d0ed40de3420ea5950c5ec34fada8 (127.12.12.130:45961): tablet 93b0559276304aee9dac11f9274058dd (table test-table [id=79156c2a014f433996cbf68950a05844]) successfully deleted
I20260429 08:00:04.410907 14083 catalog_manager.cc:5002] TS 1e3aaf4d86b744449e1df19a35e8722e (127.12.12.131:44817): tablet 93b0559276304aee9dac11f9274058dd (table test-table [id=79156c2a014f433996cbf68950a05844]) successfully deleted
W20260429 08:00:04.414578 14085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.129:45287 (error 108)
I20260429 08:00:04.414834 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 14314
I20260429 08:00:04.416090 14761 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:04.415635 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:45287 (local address 127.0.0.1:44192)
0429 08:00:04.415874 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:04.415956 (+ 82us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:45287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
W20260429 08:00:04.416221 14085 catalog_manager.cc:4729] TS 9053f40d69cc4260a8218eaa12684cbf (127.12.12.129:45287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93b0559276304aee9dac11f9274058dd: Network error: Client connection negotiation failed: client connection to 127.12.12.129:45287: connect: Connection refused (error 111)
I20260429 08:00:04.423030 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 14456
I20260429 08:00:04.429739 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 14066
2026-04-29T08:00:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7429 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:04 dist-test-slave-nhqz krb5kdc[14764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:04 dist-test-slave-nhqz krb5kdc[14764](info): set up 2 sockets
Apr 29 08:00:04 dist-test-slave-nhqz krb5kdc[14764](info): commencing operation
krb5kdc: starting...
W20260429 08:00:06.417796 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.965s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:06 dist-test-slave-nhqz krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:06.553908 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:43393
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:43393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:06.654067 14780 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:06.654314 14780 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:06.654368 14780 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:06.657706 14780 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:06.657770 14780 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:06.657794 14780 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:06.657863 14780 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:06.657893 14780 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:06.662400 14780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:43393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:06.663617 14780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:06.664492 14780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:06.670275 14788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:06.670306 14786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:06.670369 14785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:06.670567 14780 server_base.cc:1061] running on GCE node
I20260429 08:00:06.671046 14780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:06.671916 14780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:06.673089 14780 hybrid_clock.cc:648] HybridClock initialized: now 1777449606673071 us; error 39 us; skew 500 ppm
Apr 29 08:00:06 dist-test-slave-nhqz krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449606, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:06.675801 14780 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:06.676903 14780 webserver.cc:492] Webserver started at http://127.12.12.190:39707/ using document root <none> and password file <none>
I20260429 08:00:06.677454 14780 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:06.677528 14780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:06.677731 14780 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:06.679560 14780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "dab9feea4b7540478d7b1d73ae6ac863"
format_stamp: "Formatted at 2026-04-29 08:00:06 on dist-test-slave-nhqz"
server_key: "085fb429eedb0842d8cb6fd6c4453e26"
server_key_iv: "a9ea3c85c67716f29a58c91478647e42"
server_key_version: "encryptionkey@0"
I20260429 08:00:06.680037 14780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "dab9feea4b7540478d7b1d73ae6ac863"
format_stamp: "Formatted at 2026-04-29 08:00:06 on dist-test-slave-nhqz"
server_key: "085fb429eedb0842d8cb6fd6c4453e26"
server_key_iv: "a9ea3c85c67716f29a58c91478647e42"
server_key_version: "encryptionkey@0"
I20260429 08:00:06.683360 14780 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260429 08:00:06.685542 14795 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:06.686532 14780 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:00:06.686666 14780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "dab9feea4b7540478d7b1d73ae6ac863"
format_stamp: "Formatted at 2026-04-29 08:00:06 on dist-test-slave-nhqz"
server_key: "085fb429eedb0842d8cb6fd6c4453e26"
server_key_iv: "a9ea3c85c67716f29a58c91478647e42"
server_key_version: "encryptionkey@0"
I20260429 08:00:06.686772 14780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:06.699026 14780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:06.702245 14780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:06.702463 14780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:06.709734 14780 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:43393
I20260429 08:00:06.709728 14847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:43393 every 8 connection(s)
I20260429 08:00:06.710780 14780 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:06.713532 14848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:06.718570 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863: Bootstrap starting.
I20260429 08:00:06.719864 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 14780
I20260429 08:00:06.719988 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:06.720235 12338 external_mini_cluster.cc:1468] Setting key 22759e03c4f12268f2e145fcee6f140c
I20260429 08:00:06.721170 14848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:06.722005 14848 log.cc:826] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:06.723712 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863: No bootstrap required, opened a new log
I20260429 08:00:06.726272 14848 raft_consensus.cc:359] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab9feea4b7540478d7b1d73ae6ac863" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 43393 } }
I20260429 08:00:06.726490 14848 raft_consensus.cc:385] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:06.726583 14848 raft_consensus.cc:740] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab9feea4b7540478d7b1d73ae6ac863, State: Initialized, Role: FOLLOWER
I20260429 08:00:06.727084 14848 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab9feea4b7540478d7b1d73ae6ac863" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 43393 } }
I20260429 08:00:06.727233 14848 raft_consensus.cc:399] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Apr 29 08:00:06 dist-test-slave-nhqz krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:06.727320 14848 raft_consensus.cc:493] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:06.727419 14848 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:06.728533 14848 raft_consensus.cc:515] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab9feea4b7540478d7b1d73ae6ac863" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 43393 } }
I20260429 08:00:06.728927 14848 leader_election.cc:304] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab9feea4b7540478d7b1d73ae6ac863; no voters:
I20260429 08:00:06.729223 14848 leader_election.cc:290] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:06.729349 14853 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:06.729622 14853 raft_consensus.cc:697] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [term 1 LEADER]: Becoming Leader. State: Replica: dab9feea4b7540478d7b1d73ae6ac863, State: Running, Role: LEADER
I20260429 08:00:06.730006 14853 consensus_queue.cc:237] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab9feea4b7540478d7b1d73ae6ac863" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 43393 } }
I20260429 08:00:06.730450 14848 sys_catalog.cc:565] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:06.731201 14853 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dab9feea4b7540478d7b1d73ae6ac863. Latest consensus state: current_term: 1 leader_uuid: "dab9feea4b7540478d7b1d73ae6ac863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab9feea4b7540478d7b1d73ae6ac863" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 43393 } } }
I20260429 08:00:06.731384 14853 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:06.731812 14861 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:06.733196 14855 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dab9feea4b7540478d7b1d73ae6ac863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab9feea4b7540478d7b1d73ae6ac863" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 43393 } } }
I20260429 08:00:06.733295 14855 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:06.734445 14851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:06.721331 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.12.12.190:43393)
0429 08:00:06.721753 (+ 422us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:06.721763 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:06.721794 (+ 31us) server_negotiation.cc:408] Connection header received
0429 08:00:06.722366 (+ 572us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:06.722384 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:06.722632 (+ 248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:06.722973 (+ 341us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:06.723823 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:06.724569 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:06.725279 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:06.725568 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:06.727932 (+ 2364us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:06.727949 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:06.727961 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:06.727987 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:06.730837 (+ 2850us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:06.731356 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:06.731365 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:06.731373 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:06.731479 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:06.731782 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:06.731788 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:06.731791 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:06.732241 (+ 450us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:06.732440 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:06.733361 (+ 921us) server_negotiation.cc:300] Negotiation successful
0429 08:00:06.733615 (+ 254us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":147,"threads_started":1}
I20260429 08:00:06.735440 14861 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:06.740861 14861 catalog_manager.cc:1357] Generated new cluster ID: 20b88e8bc1014c2fbfa4556c0d07379d
I20260429 08:00:06.740931 14861 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:06.758563 14861 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:06.759435 14861 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:06.770114 14861 catalog_manager.cc:6044] T 00000000000000000000000000000000 P dab9feea4b7540478d7b1d73ae6ac863: Generated new TSK 0
I20260429 08:00:06.770853 14861 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:06.823943 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:43393
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:06.924695 14876 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:06.924997 14876 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:06.925094 14876 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:06.928557 14876 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:06.928623 14876 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:06.928704 14876 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:06.933598 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:06.934738 14876 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:06.935611 14876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 08:00:06.942391 14876 server_base.cc:1061] running on GCE node
W20260429 08:00:06.942337 14884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:06.942337 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:06.942333 14882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:06.943028 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:06.943562 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:06.944736 14876 hybrid_clock.cc:648] HybridClock initialized: now 1777449606944717 us; error 41 us; skew 500 ppm
Apr 29 08:00:06 dist-test-slave-nhqz krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449606, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:06.947377 14876 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:00:06.948372 14876 webserver.cc:492] Webserver started at http://127.12.12.129:41455/ using document root <none> and password file <none>
I20260429 08:00:06.949011 14876 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:06.949083 14876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:06.949275 14876 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:06.950924 14876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "361ef7e1f443419281a9a237537e8ea5"
format_stamp: "Formatted at 2026-04-29 08:00:06 on dist-test-slave-nhqz"
server_key: "d80de9f52c06431f16d5f043aa880acd"
server_key_iv: "728214dcdb83d1f7fc34acd25f96232b"
server_key_version: "encryptionkey@0"
I20260429 08:00:06.951408 14876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "361ef7e1f443419281a9a237537e8ea5"
format_stamp: "Formatted at 2026-04-29 08:00:06 on dist-test-slave-nhqz"
server_key: "d80de9f52c06431f16d5f043aa880acd"
server_key_iv: "728214dcdb83d1f7fc34acd25f96232b"
server_key_version: "encryptionkey@0"
I20260429 08:00:06.954768 14876 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:06.956773 14891 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:06.957818 14876 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:06.957953 14876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "361ef7e1f443419281a9a237537e8ea5"
format_stamp: "Formatted at 2026-04-29 08:00:06 on dist-test-slave-nhqz"
server_key: "d80de9f52c06431f16d5f043aa880acd"
server_key_iv: "728214dcdb83d1f7fc34acd25f96232b"
server_key_version: "encryptionkey@0"
I20260429 08:00:06.958057 14876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:06.973948 14876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:06.976863 14876 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:06.977087 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:06.977636 14876 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:06.978474 14876 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:06.978544 14876 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:06.978613 14876 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:06.978672 14876 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:06.987210 14876 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:43301
I20260429 08:00:06.987248 15004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:43301 every 8 connection(s)
I20260429 08:00:06.988097 14876 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:06.989518 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 14876
I20260429 08:00:06.989602 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:06.989866 12338 external_mini_cluster.cc:1468] Setting key f227c3df062c69353cffda6980a220e7
Apr 29 08:00:06 dist-test-slave-nhqz krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449606, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:00:07.000451 14851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:06.989871 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:60123 (local address 127.12.12.190:43393)
0429 08:00:06.990044 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:06.990048 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:06.990866 (+ 818us) server_negotiation.cc:408] Connection header received
0429 08:00:06.991847 (+ 981us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:06.991851 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:06.991910 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:06.991986 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:06.993476 (+ 1490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:06.993945 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:06.994578 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:06.994758 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:06.997033 (+ 2275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:06.997050 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:06.997052 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:06.997075 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:06.998576 (+ 1501us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:06.999073 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:06.999077 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:06.999079 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:06.999128 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:06.999425 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:06.999429 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:06.999431 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:06.999578 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:06.999682 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:07.000133 (+ 451us) server_negotiation.cc:300] Negotiation successful
0429 08:00:07.000259 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
I20260429 08:00:07.001214 15007 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:06.990133 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:43393 (local address 127.12.12.129:60123)
0429 08:00:06.990661 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:06.990773 (+ 112us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:06.991667 (+ 894us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:06.992144 (+ 477us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:06.992152 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:06.992485 (+ 333us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:06.993340 (+ 855us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:06.993354 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:06.994062 (+ 708us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:06.994072 (+ 10us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:06.994464 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:06.994470 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:06.994671 (+ 201us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:06.995335 (+ 664us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:06.995352 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:06.996893 (+ 1541us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:06.998705 (+ 1812us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:06.998710 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:06.998721 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:06.998975 (+ 254us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:06.999225 (+ 250us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:06.999227 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:06.999229 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:06.999332 (+ 103us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:06.999682 (+ 350us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:06.999688 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:06.999907 (+ 219us) client_negotiation.cc:770] Sending connection context
0429 08:00:07.000111 (+ 204us) client_negotiation.cc:241] Negotiation successful
0429 08:00:07.000323 (+ 212us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":131,"threads_started":1}
I20260429 08:00:07.002434 15005 heartbeater.cc:344] Connected to a master server at 127.12.12.190:43393
I20260429 08:00:07.002693 15005 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:07.003237 15005 heartbeater.cc:507] Master 127.12.12.190:43393 requested a full tablet report, sending...
I20260429 08:00:07.005004 14812 ts_manager.cc:194] Registered new tserver with Master: 361ef7e1f443419281a9a237537e8ea5 (127.12.12.129:43301)
I20260429 08:00:07.006152 14812 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:60123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:07.037982 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:43393
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:07.138422 15012 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:07.138692 15012 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:07.138795 15012 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:07.142242 15012 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:07.142340 15012 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:07.142455 15012 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:07.147310 15012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:07.148460 15012 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:07.149389 15012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:07.155750 15017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:07.155750 15020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:07.155733 15018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:07.155882 15012 server_base.cc:1061] running on GCE node
I20260429 08:00:07.156320 15012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:07.156932 15012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:07.158102 15012 hybrid_clock.cc:648] HybridClock initialized: now 1777449607158079 us; error 35 us; skew 500 ppm
Apr 29 08:00:07 dist-test-slave-nhqz krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449607, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:07.160564 15012 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:00:07.161624 15012 webserver.cc:492] Webserver started at http://127.12.12.130:43589/ using document root <none> and password file <none>
I20260429 08:00:07.162200 15012 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:07.162269 15012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:07.162477 15012 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:07.164234 15012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "dec9302c2f6a47b1bef4f0f8233ba136"
format_stamp: "Formatted at 2026-04-29 08:00:07 on dist-test-slave-nhqz"
server_key: "820d79adfc280403126982f99ef7bd9b"
server_key_iv: "325dc0c430d51cc7935740a36212c361"
server_key_version: "encryptionkey@0"
I20260429 08:00:07.164721 15012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "dec9302c2f6a47b1bef4f0f8233ba136"
format_stamp: "Formatted at 2026-04-29 08:00:07 on dist-test-slave-nhqz"
server_key: "820d79adfc280403126982f99ef7bd9b"
server_key_iv: "325dc0c430d51cc7935740a36212c361"
server_key_version: "encryptionkey@0"
I20260429 08:00:07.167824 15012 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 08:00:07.169886 15027 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:07.170779 15012 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:07.170912 15012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "dec9302c2f6a47b1bef4f0f8233ba136"
format_stamp: "Formatted at 2026-04-29 08:00:07 on dist-test-slave-nhqz"
server_key: "820d79adfc280403126982f99ef7bd9b"
server_key_iv: "325dc0c430d51cc7935740a36212c361"
server_key_version: "encryptionkey@0"
I20260429 08:00:07.171044 15012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:07.195045 15012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:07.197575 15012 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:07.197731 15012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:07.198230 15012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:07.199091 15012 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:07.199134 15012 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:07.199172 15012 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:07.199190 15012 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:07.208103 15012 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:40881
I20260429 08:00:07.208169 15140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:40881 every 8 connection(s)
I20260429 08:00:07.209115 15012 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:07.213704 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15012
I20260429 08:00:07.213800 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:07.214035 12338 external_mini_cluster.cc:1468] Setting key a8275387d6022e293843a8d3b4dd97b1
Apr 29 08:00:07 dist-test-slave-nhqz krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449607, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:07.221983 14851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:07.210932 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41987 (local address 127.12.12.190:43393)
0429 08:00:07.211080 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:07.211084 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:07.211730 (+ 646us) server_negotiation.cc:408] Connection header received
0429 08:00:07.212497 (+ 767us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:07.212500 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:07.212549 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:07.212623 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:07.214306 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.214915 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:07.215649 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.215827 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:07.218190 (+ 2363us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:07.218214 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:07.218220 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:07.218253 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:07.219987 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:07.220489 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:07.220496 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:07.220500 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:07.220559 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:07.220866 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:07.220872 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:07.220875 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:07.221070 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:07.221176 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:07.221663 (+ 487us) server_negotiation.cc:300] Negotiation successful
0429 08:00:07.221789 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 08:00:07.222492 15143 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:07.211193 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:43393 (local address 127.12.12.130:41987)
0429 08:00:07.211576 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:07.211621 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:07.212325 (+ 704us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:07.212766 (+ 441us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:07.212774 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:07.213192 (+ 418us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:07.214156 (+ 964us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:07.214171 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.215039 (+ 868us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:07.215042 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:07.215529 (+ 487us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:07.215537 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.215756 (+ 219us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:07.216386 (+ 630us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:07.216405 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:07.218028 (+ 1623us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:07.220113 (+ 2085us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:07.220118 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:07.220129 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:07.220363 (+ 234us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:07.220657 (+ 294us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:07.220660 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:07.220661 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:07.220747 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:07.221178 (+ 431us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:07.221183 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:07.221399 (+ 216us) client_negotiation.cc:770] Sending connection context
0429 08:00:07.221570 (+ 171us) client_negotiation.cc:241] Negotiation successful
0429 08:00:07.221768 (+ 198us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:00:07.223716 15141 heartbeater.cc:344] Connected to a master server at 127.12.12.190:43393
I20260429 08:00:07.223930 15141 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:07.224499 15141 heartbeater.cc:507] Master 127.12.12.190:43393 requested a full tablet report, sending...
I20260429 08:00:07.225618 14812 ts_manager.cc:194] Registered new tserver with Master: dec9302c2f6a47b1bef4f0f8233ba136 (127.12.12.130:40881)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:00:07.226302 14812 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:41987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:07.262015 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:43393
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:07.370549 15148 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:07.370862 15148 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:07.371012 15148 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:07.376976 15148 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:07.377048 15148 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:07.377126 15148 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:07.381521 15148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:07.382571 15148 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:07.383355 15148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:07.389961 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:07.389983 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:07.389984 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:07.390300 15148 server_base.cc:1061] running on GCE node
I20260429 08:00:07.390710 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:07.391309 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:07.392496 15148 hybrid_clock.cc:648] HybridClock initialized: now 1777449607392478 us; error 38 us; skew 500 ppm
Apr 29 08:00:07 dist-test-slave-nhqz krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449607, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:07.395222 15148 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:00:07.396209 15148 webserver.cc:492] Webserver started at http://127.12.12.131:36053/ using document root <none> and password file <none>
I20260429 08:00:07.396775 15148 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:07.396875 15148 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:07.397082 15148 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:07.398792 15148 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "3b689fca5a5b46d7ae3134414aabbd5d"
format_stamp: "Formatted at 2026-04-29 08:00:07 on dist-test-slave-nhqz"
server_key: "a5498061e7c64d5c16f7f5e22402955b"
server_key_iv: "7592400bfe493dbe438b72ca61ea97c3"
server_key_version: "encryptionkey@0"
I20260429 08:00:07.399288 15148 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "3b689fca5a5b46d7ae3134414aabbd5d"
format_stamp: "Formatted at 2026-04-29 08:00:07 on dist-test-slave-nhqz"
server_key: "a5498061e7c64d5c16f7f5e22402955b"
server_key_iv: "7592400bfe493dbe438b72ca61ea97c3"
server_key_version: "encryptionkey@0"
I20260429 08:00:07.402634 15148 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 08:00:07.404664 15163 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:07.405648 15148 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:07.405812 15148 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "3b689fca5a5b46d7ae3134414aabbd5d"
format_stamp: "Formatted at 2026-04-29 08:00:07 on dist-test-slave-nhqz"
server_key: "a5498061e7c64d5c16f7f5e22402955b"
server_key_iv: "7592400bfe493dbe438b72ca61ea97c3"
server_key_version: "encryptionkey@0"
I20260429 08:00:07.405913 15148 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:07.435586 15148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:07.438597 15148 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:07.438830 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:07.439404 15148 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:07.440296 15148 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:07.440397 15148 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:07.440478 15148 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:07.440532 15148 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:07.449615 15148 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:38765
I20260429 08:00:07.449651 15276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:38765 every 8 connection(s)
I20260429 08:00:07.450567 15148 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:07.457758 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15148
I20260429 08:00:07.457916 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:07.458189 12338 external_mini_cluster.cc:1468] Setting key 8f63aa4bcdec67763cdddfc80e28bf71
Apr 29 08:00:07 dist-test-slave-nhqz krb5kdc[14764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449607, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:07.464779 14851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:07.452302 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:50511 (local address 127.12.12.190:43393)
0429 08:00:07.452458 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:07.452462 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:07.453156 (+ 694us) server_negotiation.cc:408] Connection header received
0429 08:00:07.454039 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:07.454043 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:07.454103 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:07.454224 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:07.455763 (+ 1539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.456273 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:07.456986 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.457163 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:07.459914 (+ 2751us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:07.459934 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:07.459940 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:07.459973 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:07.462130 (+ 2157us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:07.462676 (+ 546us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:07.462682 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:07.462684 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:07.462739 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:07.463736 (+ 997us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:07.463739 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:07.463741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:07.463893 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:07.464002 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:07.464503 (+ 501us) server_negotiation.cc:300] Negotiation successful
0429 08:00:07.464618 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260429 08:00:07.465389 15279 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:07.452566 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:43393 (local address 127.12.12.131:50511)
0429 08:00:07.453015 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:07.453047 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:07.453849 (+ 802us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:07.454386 (+ 537us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:07.454394 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:07.454803 (+ 409us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:07.455614 (+ 811us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:07.455626 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.456416 (+ 790us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:07.456421 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:07.456868 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:07.456875 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:07.457076 (+ 201us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:07.457671 (+ 595us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:07.457694 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:07.459742 (+ 2048us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:07.462290 (+ 2548us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:07.462298 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:07.462315 (+ 17us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:07.462549 (+ 234us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:07.463532 (+ 983us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:07.463535 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:07.463536 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:07.463630 (+ 94us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:07.464001 (+ 371us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:07.464008 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:07.464252 (+ 244us) client_negotiation.cc:770] Sending connection context
0429 08:00:07.464470 (+ 218us) client_negotiation.cc:241] Negotiation successful
0429 08:00:07.464694 (+ 224us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":95,"threads_started":1}
I20260429 08:00:07.466398 15277 heartbeater.cc:344] Connected to a master server at 127.12.12.190:43393
I20260429 08:00:07.466645 15277 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:07.467134 15277 heartbeater.cc:507] Master 127.12.12.190:43393 requested a full tablet report, sending...
I20260429 08:00:07.468076 14812 ts_manager.cc:194] Registered new tserver with Master: 3b689fca5a5b46d7ae3134414aabbd5d (127.12.12.131:38765)
I20260429 08:00:07.468613 14812 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:50511
I20260429 08:00:07.472043 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 08:00:07.483529 14851 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:07.482249 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.12.12.190:43393)
0429 08:00:07.482406 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:07.482410 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:07.482425 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:00:07.482566 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:07.482569 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:07.482615 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:07.482688 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:07.483394 (+ 706us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53678: BlockingRecv error: recv got EOF from 127.0.0.1:53678 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260429 08:00:07.485044 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 14876
I20260429 08:00:07.491068 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15012
I20260429 08:00:07.497705 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15148
I20260429 08:00:07.503515 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 14780
2026-04-29T08:00:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3072 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:09 dist-test-slave-nhqz krb5kdc[15289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:09 dist-test-slave-nhqz krb5kdc[15289](info): set up 2 sockets
Apr 29 08:00:09 dist-test-slave-nhqz krb5kdc[15289](info): commencing operation
krb5kdc: starting...
W20260429 08:00:11.282094 12338 mini_kdc.cc:121] Time spent starting KDC: real 3.756s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:11 dist-test-slave-nhqz krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:11.417737 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:39303
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:39303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:11.522586 15305 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:11.522800 15305 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:11.522946 15305 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:11.526297 15305 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:11.526379 15305 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:11.526401 15305 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:11.526419 15305 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:11.526436 15305 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:11.530712 15305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:39303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:11.531823 15305 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:11.532742 15305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:11.538376 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:11.538408 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:11.538414 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:11.538558 15305 server_base.cc:1061] running on GCE node
I20260429 08:00:11.539129 15305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:11.540036 15305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:11.541214 15305 hybrid_clock.cc:648] HybridClock initialized: now 1777449611541192 us; error 56 us; skew 500 ppm
Apr 29 08:00:11 dist-test-slave-nhqz krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449611, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:11.544035 15305 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:11.545215 15305 webserver.cc:492] Webserver started at http://127.12.12.190:46631/ using document root <none> and password file <none>
I20260429 08:00:11.545847 15305 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:11.545922 15305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:11.546144 15305 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:11.547905 15305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "913d305e96b843358dcdd07a311f0f2e"
format_stamp: "Formatted at 2026-04-29 08:00:11 on dist-test-slave-nhqz"
server_key: "e0e1c4051b8351c793b8baf00a52b580"
server_key_iv: "d81dd0fb5982fb1df94530b1b55b576c"
server_key_version: "encryptionkey@0"
I20260429 08:00:11.548413 15305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "913d305e96b843358dcdd07a311f0f2e"
format_stamp: "Formatted at 2026-04-29 08:00:11 on dist-test-slave-nhqz"
server_key: "e0e1c4051b8351c793b8baf00a52b580"
server_key_iv: "d81dd0fb5982fb1df94530b1b55b576c"
server_key_version: "encryptionkey@0"
I20260429 08:00:11.551923 15305 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 08:00:11.554176 15320 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:11.555114 15305 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 08:00:11.555254 15305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "913d305e96b843358dcdd07a311f0f2e"
format_stamp: "Formatted at 2026-04-29 08:00:11 on dist-test-slave-nhqz"
server_key: "e0e1c4051b8351c793b8baf00a52b580"
server_key_iv: "d81dd0fb5982fb1df94530b1b55b576c"
server_key_version: "encryptionkey@0"
I20260429 08:00:11.555364 15305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:11.571070 15305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:11.578969 15305 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:11.579224 15305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:11.586978 15372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:39303 every 8 connection(s)
I20260429 08:00:11.586983 15305 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:39303
I20260429 08:00:11.588035 15305 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:11.590840 15373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:11.593223 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15305
I20260429 08:00:11.593367 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:11.593609 12338 external_mini_cluster.cc:1468] Setting key cacbee2f31a97bedb99290da20789faa
I20260429 08:00:11.597026 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e: Bootstrap starting.
I20260429 08:00:11.599354 15373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:11.600137 15373 log.cc:826] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e: Log is configured to *not* fsync() on all Append() calls
Apr 29 08:00:11 dist-test-slave-nhqz krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:11.602304 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e: No bootstrap required, opened a new log
I20260429 08:00:11.605022 15373 raft_consensus.cc:359] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913d305e96b843358dcdd07a311f0f2e" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39303 } }
I20260429 08:00:11.605252 15373 raft_consensus.cc:385] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:11.605386 15373 raft_consensus.cc:740] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913d305e96b843358dcdd07a311f0f2e, State: Initialized, Role: FOLLOWER
I20260429 08:00:11.605878 15373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913d305e96b843358dcdd07a311f0f2e" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39303 } }
I20260429 08:00:11.606033 15373 raft_consensus.cc:399] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:11.606106 15373 raft_consensus.cc:493] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:11.606108 15376 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:11.594890 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.12.12.190:39303)
0429 08:00:11.595374 (+ 484us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:11.595385 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:11.595419 (+ 34us) server_negotiation.cc:408] Connection header received
0429 08:00:11.595952 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:11.595968 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:11.596260 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:11.596577 (+ 317us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:11.597658 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:11.598405 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:11.599070 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:11.599380 (+ 310us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:11.601762 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:11.601790 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:11.601802 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:11.601830 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:11.603638 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:11.604073 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:11.604078 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:11.604082 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:11.604157 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:11.604399 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:11.604402 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:11.604404 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:11.604702 (+ 298us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:11.604928 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:11.605211 (+ 283us) server_negotiation.cc:300] Negotiation successful
0429 08:00:11.605466 (+ 255us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":182,"threads_started":1}
I20260429 08:00:11.606230 15373 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:11.607316 15373 raft_consensus.cc:515] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913d305e96b843358dcdd07a311f0f2e" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39303 } }
I20260429 08:00:11.607671 15373 leader_election.cc:304] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913d305e96b843358dcdd07a311f0f2e; no voters:
I20260429 08:00:11.607961 15373 leader_election.cc:290] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:11.608098 15378 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:11.608330 15378 raft_consensus.cc:697] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [term 1 LEADER]: Becoming Leader. State: Replica: 913d305e96b843358dcdd07a311f0f2e, State: Running, Role: LEADER
I20260429 08:00:11.608718 15378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913d305e96b843358dcdd07a311f0f2e" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39303 } }
I20260429 08:00:11.609184 15373 sys_catalog.cc:565] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:11.610365 15379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913d305e96b843358dcdd07a311f0f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913d305e96b843358dcdd07a311f0f2e" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39303 } } }
I20260429 08:00:11.610404 15380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913d305e96b843358dcdd07a311f0f2e. Latest consensus state: current_term: 1 leader_uuid: "913d305e96b843358dcdd07a311f0f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913d305e96b843358dcdd07a311f0f2e" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39303 } } }
I20260429 08:00:11.610481 15379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:11.610487 15380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:11.610859 15382 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:11.613880 15382 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:11.619108 15382 catalog_manager.cc:1357] Generated new cluster ID: af5ae8f8c59a4faa9115f0af257283a1
I20260429 08:00:11.619192 15382 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:11.629575 15382 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:11.630628 15382 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:11.641587 15382 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 913d305e96b843358dcdd07a311f0f2e: Generated new TSK 0
I20260429 08:00:11.642186 15382 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:11.701763 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:39303
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:11.809242 15401 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:11.809450 15401 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:11.809502 15401 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:11.812762 15401 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:11.812830 15401 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:11.812918 15401 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:11.817078 15401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:11.817999 15401 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:11.818748 15401 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:11.824924 15409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:11.824956 15406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:11.824957 15407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:11.825301 15401 server_base.cc:1061] running on GCE node
I20260429 08:00:11.825713 15401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:11.826256 15401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:11.827435 15401 hybrid_clock.cc:648] HybridClock initialized: now 1777449611827411 us; error 42 us; skew 500 ppm
Apr 29 08:00:11 dist-test-slave-nhqz krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449611, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:11.829962 15401 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:00:11.830909 15401 webserver.cc:492] Webserver started at http://127.12.12.129:36067/ using document root <none> and password file <none>
I20260429 08:00:11.831441 15401 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:11.831514 15401 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:11.831708 15401 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:11.833348 15401 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "fb7f9f4f04a840d680f4aa6277f504b3"
format_stamp: "Formatted at 2026-04-29 08:00:11 on dist-test-slave-nhqz"
server_key: "b0aa8731ef41e0de720a1413719649b0"
server_key_iv: "25d5bd3f7e101afd3359c273c010aaa0"
server_key_version: "encryptionkey@0"
I20260429 08:00:11.833834 15401 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "fb7f9f4f04a840d680f4aa6277f504b3"
format_stamp: "Formatted at 2026-04-29 08:00:11 on dist-test-slave-nhqz"
server_key: "b0aa8731ef41e0de720a1413719649b0"
server_key_iv: "25d5bd3f7e101afd3359c273c010aaa0"
server_key_version: "encryptionkey@0"
I20260429 08:00:11.836885 15401 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:00:11.838869 15416 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:11.839788 15401 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:11.839933 15401 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "fb7f9f4f04a840d680f4aa6277f504b3"
format_stamp: "Formatted at 2026-04-29 08:00:11 on dist-test-slave-nhqz"
server_key: "b0aa8731ef41e0de720a1413719649b0"
server_key_iv: "25d5bd3f7e101afd3359c273c010aaa0"
server_key_version: "encryptionkey@0"
I20260429 08:00:11.840031 15401 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:11.872927 15401 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:11.875509 15401 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:11.875715 15401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:11.876329 15401 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:11.877256 15401 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:11.877331 15401 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:11.877398 15401 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:11.877442 15401 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:11.886651 15401 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:33533
I20260429 08:00:11.886689 15529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:33533 every 8 connection(s)
I20260429 08:00:11.887569 15401 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:11.888278 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15401
I20260429 08:00:11.888401 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:11.888634 12338 external_mini_cluster.cc:1468] Setting key 9a80ad1bc56bcaf458203e395bbc639a
Apr 29 08:00:11 dist-test-slave-nhqz krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449611, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
I20260429 08:00:11.900576 15376 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:11.889472 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:44993 (local address 127.12.12.190:39303)
0429 08:00:11.889951 (+ 479us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:11.889956 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:11.890499 (+ 543us) server_negotiation.cc:408] Connection header received
0429 08:00:11.891319 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:11.891326 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:11.891401 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:11.891502 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:11.892974 (+ 1472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:11.893806 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:11.894425 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:11.894661 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:11.897201 (+ 2540us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:11.897226 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:11.897232 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:11.897268 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:11.898688 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:11.899228 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:11.899231 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:11.899232 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:11.899271 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:11.899577 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:11.899579 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:11.899580 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:11.899725 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:11.899799 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:11.900339 (+ 540us) server_negotiation.cc:300] Negotiation successful
0429 08:00:11.900455 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260429 08:00:11.901517 15532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:11.889749 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:39303 (local address 127.12.12.129:44993)
0429 08:00:11.890338 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:11.890386 (+ 48us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:11.891122 (+ 736us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:11.891708 (+ 586us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:11.891721 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:11.892318 (+ 597us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:11.892776 (+ 458us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:11.892787 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:11.893922 (+ 1135us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:11.893925 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:11.894306 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:11.894312 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:11.894464 (+ 152us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:11.895496 (+ 1032us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:11.895514 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:11.897033 (+ 1519us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:11.898812 (+ 1779us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:11.898818 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:11.898849 (+ 31us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:11.899123 (+ 274us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:11.899359 (+ 236us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:11.899361 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:11.899363 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:11.899473 (+ 110us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:11.899811 (+ 338us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:11.899815 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:11.900040 (+ 225us) client_negotiation.cc:770] Sending connection context
0429 08:00:11.900284 (+ 244us) client_negotiation.cc:241] Negotiation successful
0429 08:00:11.900543 (+ 259us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":111,"threads_started":1}
I20260429 08:00:11.902755 15530 heartbeater.cc:344] Connected to a master server at 127.12.12.190:39303
I20260429 08:00:11.903051 15530 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:11.903564 15530 heartbeater.cc:507] Master 127.12.12.190:39303 requested a full tablet report, sending...
I20260429 08:00:11.905278 15337 ts_manager.cc:194] Registered new tserver with Master: fb7f9f4f04a840d680f4aa6277f504b3 (127.12.12.129:33533)
I20260429 08:00:11.906520 15337 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:44993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:11.940165 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:39303
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:12.041802 15537 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:12.042052 15537 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:12.042172 15537 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:12.045624 15537 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:12.045683 15537 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:12.045806 15537 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:12.050030 15537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:12.051118 15537 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:12.051982 15537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:12.058578 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:12.058640 15545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:12.058585 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:12.058957 15537 server_base.cc:1061] running on GCE node
I20260429 08:00:12.059336 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:12.059921 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:12.061090 15537 hybrid_clock.cc:648] HybridClock initialized: now 1777449612061072 us; error 42 us; skew 500 ppm
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449612, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:12.063827 15537 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:00:12.064901 15537 webserver.cc:492] Webserver started at http://127.12.12.130:33177/ using document root <none> and password file <none>
I20260429 08:00:12.065479 15537 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:12.065526 15537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:12.065680 15537 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:12.067229 15537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "e3d5b6f831f8466f995e4af887bccbfe"
format_stamp: "Formatted at 2026-04-29 08:00:12 on dist-test-slave-nhqz"
server_key: "418a60997672ddbb90f37c90f42dbb3b"
server_key_iv: "3ce7a28ab60b48985e01d93c5603a87a"
server_key_version: "encryptionkey@0"
I20260429 08:00:12.067644 15537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "e3d5b6f831f8466f995e4af887bccbfe"
format_stamp: "Formatted at 2026-04-29 08:00:12 on dist-test-slave-nhqz"
server_key: "418a60997672ddbb90f37c90f42dbb3b"
server_key_iv: "3ce7a28ab60b48985e01d93c5603a87a"
server_key_version: "encryptionkey@0"
I20260429 08:00:12.070801 15537 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:12.072902 15552 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:12.073875 15537 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:12.073966 15537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "e3d5b6f831f8466f995e4af887bccbfe"
format_stamp: "Formatted at 2026-04-29 08:00:12 on dist-test-slave-nhqz"
server_key: "418a60997672ddbb90f37c90f42dbb3b"
server_key_iv: "3ce7a28ab60b48985e01d93c5603a87a"
server_key_version: "encryptionkey@0"
I20260429 08:00:12.074043 15537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:12.089819 15537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:12.092720 15537 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:12.092922 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:12.093456 15537 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:12.094274 15537 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:12.094319 15537 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:12.094365 15537 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:12.094388 15537 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:12.103264 15537 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:38247
I20260429 08:00:12.103288 15665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:38247 every 8 connection(s)
I20260429 08:00:12.104193 15537 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:12.105978 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15537
I20260429 08:00:12.106087 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:12.106344 12338 external_mini_cluster.cc:1468] Setting key 6ba04ab35c58f791bad956bade079111
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449612, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:00:12.117060 15376 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:12.105912 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:36813 (local address 127.12.12.190:39303)
0429 08:00:12.106080 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:12.106086 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:12.106913 (+ 827us) server_negotiation.cc:408] Connection header received
0429 08:00:12.107891 (+ 978us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:12.107894 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:12.107947 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:12.108026 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:12.109158 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.109830 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:12.110509 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.110735 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:12.113602 (+ 2867us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:12.113627 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:12.113630 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:12.113653 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:12.115310 (+ 1657us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:12.115759 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:12.115762 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:12.115763 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:12.115806 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:12.116043 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:12.116045 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:12.116046 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:12.116246 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:12.116319 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:12.116761 (+ 442us) server_negotiation.cc:300] Negotiation successful
0429 08:00:12.116934 (+ 173us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:00:12.117887 15668 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:12.106296 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:39303 (local address 127.12.12.130:36813)
0429 08:00:12.106757 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:12.106797 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:12.107644 (+ 847us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:12.108166 (+ 522us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:12.108173 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:12.108534 (+ 361us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:12.109019 (+ 485us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:12.109031 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.109951 (+ 920us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:12.109958 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:12.110390 (+ 432us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:12.110397 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.110544 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:12.111714 (+ 1170us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:12.111746 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:12.113456 (+ 1710us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:12.115433 (+ 1977us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:12.115438 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:12.115448 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:12.115664 (+ 216us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:12.115889 (+ 225us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:12.115891 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:12.115892 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:12.115968 (+ 76us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:12.116336 (+ 368us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:12.116340 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:12.116555 (+ 215us) client_negotiation.cc:770] Sending connection context
0429 08:00:12.116719 (+ 164us) client_negotiation.cc:241] Negotiation successful
0429 08:00:12.116988 (+ 269us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260429 08:00:12.119035 15666 heartbeater.cc:344] Connected to a master server at 127.12.12.190:39303
I20260429 08:00:12.119305 15666 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:12.119918 15666 heartbeater.cc:507] Master 127.12.12.190:39303 requested a full tablet report, sending...
I20260429 08:00:12.121085 15337 ts_manager.cc:194] Registered new tserver with Master: e3d5b6f831f8466f995e4af887bccbfe (127.12.12.130:38247)
I20260429 08:00:12.121903 15337 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:36813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:12.162444 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:39303
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:12.272034 15673 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:12.272267 15673 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:12.272320 15673 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:12.275710 15673 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:12.275774 15673 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:12.275848 15673 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:12.280156 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:12.281284 15673 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:12.282136 15673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:12.288568 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:12.288599 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:12.288697 15673 server_base.cc:1061] running on GCE node
W20260429 08:00:12.288908 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:12.289315 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:12.289842 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:12.291011 15673 hybrid_clock.cc:648] HybridClock initialized: now 1777449612290992 us; error 42 us; skew 500 ppm
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449612, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:12.293843 15673 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:00:12.295010 15673 webserver.cc:492] Webserver started at http://127.12.12.131:39001/ using document root <none> and password file <none>
I20260429 08:00:12.295586 15673 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:12.295635 15673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:12.295866 15673 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:12.297688 15673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "d40149915a924b8fba8734986cd69ba5"
format_stamp: "Formatted at 2026-04-29 08:00:12 on dist-test-slave-nhqz"
server_key: "a8b500ac579ff1c8209b1a272f75c5e9"
server_key_iv: "622e161750b26bcd989be09647b18f17"
server_key_version: "encryptionkey@0"
I20260429 08:00:12.298189 15673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "d40149915a924b8fba8734986cd69ba5"
format_stamp: "Formatted at 2026-04-29 08:00:12 on dist-test-slave-nhqz"
server_key: "a8b500ac579ff1c8209b1a272f75c5e9"
server_key_iv: "622e161750b26bcd989be09647b18f17"
server_key_version: "encryptionkey@0"
I20260429 08:00:12.301669 15673 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:00:12.303879 15688 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:12.304905 15673 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:00:12.305027 15673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "d40149915a924b8fba8734986cd69ba5"
format_stamp: "Formatted at 2026-04-29 08:00:12 on dist-test-slave-nhqz"
server_key: "a8b500ac579ff1c8209b1a272f75c5e9"
server_key_iv: "622e161750b26bcd989be09647b18f17"
server_key_version: "encryptionkey@0"
I20260429 08:00:12.305136 15673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:12.315891 15673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:12.318568 15673 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:12.318790 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:12.319365 15673 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:12.320245 15673 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:12.320317 15673 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:12.320392 15673 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:12.320437 15673 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:12.329968 15673 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:35793
I20260429 08:00:12.329989 15801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:35793 every 8 connection(s)
I20260429 08:00:12.331012 15673 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449612, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:12.339313 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15673
I20260429 08:00:12.339442 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:12.339682 12338 external_mini_cluster.cc:1468] Setting key 829f2a867db5dbe20ab1300d055fefc3
I20260429 08:00:12.343442 15376 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:12.332881 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:36587 (local address 127.12.12.190:39303)
0429 08:00:12.333027 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:12.333030 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:12.333655 (+ 625us) server_negotiation.cc:408] Connection header received
0429 08:00:12.334437 (+ 782us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:12.334440 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:12.334503 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:12.334578 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:12.335742 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.336250 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:12.336918 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.337060 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:12.339779 (+ 2719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:12.339805 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:12.339807 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:12.339832 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:12.341420 (+ 1588us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:12.342017 (+ 597us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:12.342021 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:12.342023 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:12.342074 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:12.342386 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:12.342389 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:12.342391 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:12.342540 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:12.342644 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:12.343144 (+ 500us) server_negotiation.cc:300] Negotiation successful
0429 08:00:12.343263 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:00:12.344134 15804 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:12.333135 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:39303 (local address 127.12.12.131:36587)
0429 08:00:12.333513 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:12.333545 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:12.334251 (+ 706us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:12.334752 (+ 501us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:12.334761 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:12.335145 (+ 384us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:12.335612 (+ 467us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:12.335623 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.336363 (+ 740us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:12.336365 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:12.336750 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:12.336756 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:12.336997 (+ 241us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:12.338043 (+ 1046us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:12.338061 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:12.339614 (+ 1553us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:12.341543 (+ 1929us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:12.341550 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:12.341562 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:12.341911 (+ 349us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:12.342182 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:12.342185 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:12.342188 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:12.342291 (+ 103us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:12.342643 (+ 352us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:12.342650 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:12.342904 (+ 254us) client_negotiation.cc:770] Sending connection context
0429 08:00:12.343116 (+ 212us) client_negotiation.cc:241] Negotiation successful
0429 08:00:12.343346 (+ 230us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260429 08:00:12.345130 15802 heartbeater.cc:344] Connected to a master server at 127.12.12.190:39303
I20260429 08:00:12.345378 15802 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:12.345880 15802 heartbeater.cc:507] Master 127.12.12.190:39303 requested a full tablet report, sending...
I20260429 08:00:12.346901 15337 ts_manager.cc:194] Registered new tserver with Master: d40149915a924b8fba8734986cd69ba5 (127.12.12.131:35793)
I20260429 08:00:12.347450 15337 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:36587
I20260429 08:00:12.353451 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 08:00:12.482939 15376 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:12.479494 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.12.12.190:39303)
0429 08:00:12.479632 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:12.479636 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:12.480450 (+ 814us) server_negotiation.cc:408] Connection header received
0429 08:00:12.481435 (+ 985us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:12.481439 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:12.481502 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:12.481608 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:12.482787 (+ 1179us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48956: BlockingRecv error: recv got EOF from 127.0.0.1:48956 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260429 08:00:12.484593 15376 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:12.483673 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.12.12.190:39303)
0429 08:00:12.483811 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:12.483815 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:12.483829 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:00:12.483991 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:12.483997 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:12.484052 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:12.484155 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:12.484502 (+ 347us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48966: BlockingRecv error: recv got EOF from 127.0.0.1:48966 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260429 08:00:12.485607 15376 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:12.484918 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.12.12.190:39303)
0429 08:00:12.485037 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:12.485040 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:12.485052 (+ 12us) server_negotiation.cc:408] Connection header received
0429 08:00:12.485120 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:12.485122 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:12.485158 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:12.485223 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:12.485527 (+ 304us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48978: BlockingRecv error: recv got EOF from 127.0.0.1:48978 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260429 08:00:12.489398 15376 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:12.488640 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.12.12.190:39303)
0429 08:00:12.488756 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:12.488760 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:12.488772 (+ 12us) server_negotiation.cc:408] Connection header received
0429 08:00:12.488897 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:12.488899 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:12.488935 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:12.488999 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:12.489319 (+ 320us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48990: BlockingRecv error: recv got EOF from 127.0.0.1:48990 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:12.548528 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15401
I20260429 08:00:12.554200 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15537
I20260429 08:00:12.559597 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15673
I20260429 08:00:12.567070 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15305
2026-04-29T08:00:12Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (5063 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15825](info): set up 2 sockets
Apr 29 08:00:12 dist-test-slave-nhqz krb5kdc[15825](info): commencing operation
krb5kdc: starting...
W20260429 08:00:14.546069 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.956s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:14 dist-test-slave-nhqz krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:14.683436 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:39189
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:14.787098 15841 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:14.787377 15841 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:14.787475 15841 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:14.790915 15841 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:14.791002 15841 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:14.791049 15841 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:14.791122 15841 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:14.791164 15841 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:14.795322 15841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:14.796440 15841 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:14.797309 15841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:14.802582 15849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:14.802592 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:14.802582 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:14.802922 15841 server_base.cc:1061] running on GCE node
I20260429 08:00:14.803381 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:14.804224 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:14.805377 15841 hybrid_clock.cc:648] HybridClock initialized: now 1777449614805359 us; error 30 us; skew 500 ppm
Apr 29 08:00:14 dist-test-slave-nhqz krb5kdc[15825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449614, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:14.807821 15841 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:14.809008 15841 webserver.cc:492] Webserver started at http://127.12.12.190:34095/ using document root <none> and password file <none>
I20260429 08:00:14.809549 15841 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:14.809623 15841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:14.809823 15841 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:14.811476 15841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "b1aeda8a9a7741e59b51ee798ec77147"
format_stamp: "Formatted at 2026-04-29 08:00:14 on dist-test-slave-nhqz"
server_key: "beca657f42b8bbe23be215e2ffbb5a07"
server_key_iv: "53dd42435d226f4edf5fe848d939fe9d"
server_key_version: "encryptionkey@0"
I20260429 08:00:14.811942 15841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "b1aeda8a9a7741e59b51ee798ec77147"
format_stamp: "Formatted at 2026-04-29 08:00:14 on dist-test-slave-nhqz"
server_key: "beca657f42b8bbe23be215e2ffbb5a07"
server_key_iv: "53dd42435d226f4edf5fe848d939fe9d"
server_key_version: "encryptionkey@0"
I20260429 08:00:14.815192 15841 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:14.817287 15856 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:14.818203 15841 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:14.818342 15841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "b1aeda8a9a7741e59b51ee798ec77147"
format_stamp: "Formatted at 2026-04-29 08:00:14 on dist-test-slave-nhqz"
server_key: "beca657f42b8bbe23be215e2ffbb5a07"
server_key_iv: "53dd42435d226f4edf5fe848d939fe9d"
server_key_version: "encryptionkey@0"
I20260429 08:00:14.818478 15841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:14.847635 15841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:14.850374 15841 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:14.850598 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:14.857666 15841 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:39189
I20260429 08:00:14.857682 15908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:39189 every 8 connection(s)
I20260429 08:00:14.858625 15841 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:14.859058 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15841
I20260429 08:00:14.859206 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:14.859468 12338 external_mini_cluster.cc:1468] Setting key 94e04f55689291c811c83fc8d591702d
I20260429 08:00:14.861871 15909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:14.867688 15909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147: Bootstrap starting.
Apr 29 08:00:14 dist-test-slave-nhqz krb5kdc[15825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:14.869923 15909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:14.870770 15909 log.cc:826] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:14.872773 15909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147: No bootstrap required, opened a new log
I20260429 08:00:14.872951 15912 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:14.860535 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.12.12.190:39189)
0429 08:00:14.860918 (+ 383us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:14.860928 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:14.860982 (+ 54us) server_negotiation.cc:408] Connection header received
0429 08:00:14.862364 (+ 1382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:14.862387 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:14.862712 (+ 325us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:14.863071 (+ 359us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:14.863888 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:14.865163 (+ 1275us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:14.865833 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:14.866129 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:14.868388 (+ 2259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:14.868442 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:14.868454 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:14.868484 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:14.870386 (+ 1902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:14.870864 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:14.870873 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:14.870880 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:14.870962 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:14.871242 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:14.871248 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:14.871250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:14.871645 (+ 395us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:14.871779 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:14.872081 (+ 302us) server_negotiation.cc:300] Negotiation successful
0429 08:00:14.872329 (+ 248us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260429 08:00:14.875564 12338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 08:00:14.875808 15909 raft_consensus.cc:359] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1aeda8a9a7741e59b51ee798ec77147" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39189 } }
I20260429 08:00:14.875977 15909 raft_consensus.cc:385] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:14.876052 15909 raft_consensus.cc:740] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1aeda8a9a7741e59b51ee798ec77147, State: Initialized, Role: FOLLOWER
I20260429 08:00:14.876461 15909 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1aeda8a9a7741e59b51ee798ec77147" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39189 } }
I20260429 08:00:14.876564 15909 raft_consensus.cc:399] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:14.876639 15909 raft_consensus.cc:493] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:14.876719 15909 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:14.877645 15909 raft_consensus.cc:515] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1aeda8a9a7741e59b51ee798ec77147" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39189 } }
I20260429 08:00:14.878026 15909 leader_election.cc:304] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1aeda8a9a7741e59b51ee798ec77147; no voters:
I20260429 08:00:14.878353 15909 leader_election.cc:290] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:14.878492 15915 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:14.878738 15915 raft_consensus.cc:697] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [term 1 LEADER]: Becoming Leader. State: Replica: b1aeda8a9a7741e59b51ee798ec77147, State: Running, Role: LEADER
I20260429 08:00:14.879061 15915 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1aeda8a9a7741e59b51ee798ec77147" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39189 } }
I20260429 08:00:14.879561 15909 sys_catalog.cc:565] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:14.880148 15915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1aeda8a9a7741e59b51ee798ec77147. Latest consensus state: current_term: 1 leader_uuid: "b1aeda8a9a7741e59b51ee798ec77147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1aeda8a9a7741e59b51ee798ec77147" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39189 } } }
I20260429 08:00:14.880366 15915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:14.880513 15917 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1aeda8a9a7741e59b51ee798ec77147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1aeda8a9a7741e59b51ee798ec77147" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 39189 } } }
I20260429 08:00:14.880595 15917 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:14.880873 15919 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:14.884035 15919 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:14.889654 15919 catalog_manager.cc:1357] Generated new cluster ID: 52cbef9fea274aad902af75c2719d166
I20260429 08:00:14.889792 15919 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:14.896224 15912 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:14.891559 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.12.12.190:39189)
0429 08:00:14.891692 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:14.891696 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:14.891948 (+ 252us) server_negotiation.cc:408] Connection header received
0429 08:00:14.892065 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:14.892069 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:14.892137 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:14.892231 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:14.892958 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:14.893710 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:14.894528 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:14.894740 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:14.895180 (+ 440us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:14.895197 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:14.895200 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:14.895727 (+ 527us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:14.895763 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:14.895780 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:14.895894 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:14.896005 (+ 111us) server_negotiation.cc:300] Negotiation successful
0429 08:00:14.896056 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 08:00:14.913648 15919 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:14.914492 15919 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:14.923659 15919 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b1aeda8a9a7741e59b51ee798ec77147: Generated new TSK 0
I20260429 08:00:14.924314 15919 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:00:14.938059 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15841
2026-04-29T08:00:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2369 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:14 dist-test-slave-nhqz krb5kdc[15942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:14 dist-test-slave-nhqz krb5kdc[15942](info): set up 2 sockets
Apr 29 08:00:14 dist-test-slave-nhqz krb5kdc[15942](info): commencing operation
krb5kdc: starting...
W20260429 08:00:16.937978 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:17.080099 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:37215
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:17.181268 15958 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:17.181550 15958 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:17.181643 15958 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:17.184991 15958 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:17.185096 15958 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:17.185155 15958 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:17.185216 15958 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:17.185261 15958 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:17.189709 15958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:17.191025 15958 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:17.191828 15958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:17.197183 15966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:17.197250 15963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:17.197268 15964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:17.197942 15958 server_base.cc:1061] running on GCE node
I20260429 08:00:17.198419 15958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:17.199299 15958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:17.200495 15958 hybrid_clock.cc:648] HybridClock initialized: now 1777449617200470 us; error 41 us; skew 500 ppm
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:17.203290 15958 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:17.204349 15958 webserver.cc:492] Webserver started at http://127.12.12.190:34033/ using document root <none> and password file <none>
I20260429 08:00:17.204953 15958 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:17.205026 15958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:17.205227 15958 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:17.206848 15958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "8ad52cf19fe84f6695ea32b32130b958"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "4f31e0bd5d5eb7c0272fc84dcb690c63"
server_key_iv: "4ef3996e96d21d44da07da358f793710"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.207360 15958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "8ad52cf19fe84f6695ea32b32130b958"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "4f31e0bd5d5eb7c0272fc84dcb690c63"
server_key_iv: "4ef3996e96d21d44da07da358f793710"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.210637 15958 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:17.212760 15973 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.213829 15958 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:17.213981 15958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "8ad52cf19fe84f6695ea32b32130b958"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "4f31e0bd5d5eb7c0272fc84dcb690c63"
server_key_iv: "4ef3996e96d21d44da07da358f793710"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.214095 15958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:17.223263 15958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:17.229200 15958 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:17.229423 15958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:17.237118 15958 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:37215
I20260429 08:00:17.237133 16025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:37215 every 8 connection(s)
I20260429 08:00:17.238214 15958 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:17.240934 16026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:17.245291 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 15958
I20260429 08:00:17.245448 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:17.245683 12338 external_mini_cluster.cc:1468] Setting key 651bca9777749dea0d05e267e1432649
I20260429 08:00:17.246287 16026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958: Bootstrap starting.
I20260429 08:00:17.248734 16026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:17.249534 16026 log.cc:826] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:17.251413 16026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958: No bootstrap required, opened a new log
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:17.254606 16026 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad52cf19fe84f6695ea32b32130b958" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37215 } }
I20260429 08:00:17.254781 16026 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:17.254817 16026 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad52cf19fe84f6695ea32b32130b958, State: Initialized, Role: FOLLOWER
I20260429 08:00:17.255268 16026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad52cf19fe84f6695ea32b32130b958" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37215 } }
I20260429 08:00:17.255393 16026 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:17.255487 16026 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:17.255609 16026 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:17.256543 16026 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad52cf19fe84f6695ea32b32130b958" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37215 } }
I20260429 08:00:17.256917 16026 leader_election.cc:304] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ad52cf19fe84f6695ea32b32130b958; no voters:
I20260429 08:00:17.257215 16026 leader_election.cc:290] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:17.257537 16031 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:17.257766 16031 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad52cf19fe84f6695ea32b32130b958, State: Running, Role: LEADER
I20260429 08:00:17.258047 16031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad52cf19fe84f6695ea32b32130b958" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37215 } }
I20260429 08:00:17.258412 16026 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:17.259052 16029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.246951 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.12.12.190:37215)
0429 08:00:17.247404 (+ 453us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:17.247416 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:17.247461 (+ 45us) server_negotiation.cc:408] Connection header received
0429 08:00:17.248117 (+ 656us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:17.248139 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:17.248484 (+ 345us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:17.248856 (+ 372us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:17.249675 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.250694 (+ 1019us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.251359 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.251981 (+ 622us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.253968 (+ 1987us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:17.253985 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:17.253996 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:17.254058 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:17.256035 (+ 1977us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.256511 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.256516 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.256520 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.256584 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.256888 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.256891 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.256892 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.257327 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:17.257541 (+ 214us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:17.257925 (+ 384us) server_negotiation.cc:300] Negotiation successful
0429 08:00:17.258224 (+ 299us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":130,"threads_started":1}
I20260429 08:00:17.259462 16033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ad52cf19fe84f6695ea32b32130b958. Latest consensus state: current_term: 1 leader_uuid: "8ad52cf19fe84f6695ea32b32130b958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad52cf19fe84f6695ea32b32130b958" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37215 } } }
I20260429 08:00:17.259560 16033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:17.259809 16032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ad52cf19fe84f6695ea32b32130b958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad52cf19fe84f6695ea32b32130b958" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37215 } } }
I20260429 08:00:17.259902 16032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958 [sys.catalog]: This master's current role is: LEADER
W20260429 08:00:17.263228 16046 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 08:00:17.263324 16046 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 08:00:17.263401 16040 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:17.264111 16040 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:17.268878 16040 catalog_manager.cc:1357] Generated new cluster ID: 1250779a00694e5381d89df40a35aacf
I20260429 08:00:17.268945 16040 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:17.291093 16040 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:17.291942 16040 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:17.297298 16040 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 8ad52cf19fe84f6695ea32b32130b958: Generated new TSK 0
I20260429 08:00:17.297835 16040 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:17.350314 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:37215
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:17.450730 16054 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:17.450937 16054 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:17.450991 16054 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:17.454125 16054 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:17.454178 16054 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:17.454316 16054 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:17.458415 16054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:17.459510 16054 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:17.460413 16054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:17.466722 16060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:17.466879 16054 server_base.cc:1061] running on GCE node
W20260429 08:00:17.466723 16062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:17.466722 16059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:17.467308 16054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:17.467825 16054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:17.468997 16054 hybrid_clock.cc:648] HybridClock initialized: now 1777449617468978 us; error 37 us; skew 500 ppm
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:17.471525 16054 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:00:17.472463 16054 webserver.cc:492] Webserver started at http://127.12.12.129:43387/ using document root <none> and password file <none>
I20260429 08:00:17.473086 16054 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:17.473193 16054 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:17.473416 16054 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:17.475006 16054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "6211273f1790496394d30daa7151485f"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "d0e32a9033b2ed5b7c1780e1fb78307a"
server_key_iv: "7b0aa2b66c642d0466c035998e2cc997"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.475482 16054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "6211273f1790496394d30daa7151485f"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "d0e32a9033b2ed5b7c1780e1fb78307a"
server_key_iv: "7b0aa2b66c642d0466c035998e2cc997"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.478781 16054 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:17.480906 16069 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.481832 16054 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:17.481963 16054 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "6211273f1790496394d30daa7151485f"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "d0e32a9033b2ed5b7c1780e1fb78307a"
server_key_iv: "7b0aa2b66c642d0466c035998e2cc997"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.482064 16054 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:17.494544 16054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:17.496951 16054 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:17.497135 16054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:17.497681 16054 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:17.498497 16054 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:17.498569 16054 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.498636 16054 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:17.498684 16054 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.507701 16054 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:36879
I20260429 08:00:17.507720 16182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:36879 every 8 connection(s)
I20260429 08:00:17.508617 16054 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:17.515870 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16054
I20260429 08:00:17.516018 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:17.516289 12338 external_mini_cluster.cc:1468] Setting key fac900ba1998c771563daacbd1521a50
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:17.520962 16029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.510259 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:52553 (local address 127.12.12.190:37215)
0429 08:00:17.510413 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:17.510417 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:17.511040 (+ 623us) server_negotiation.cc:408] Connection header received
0429 08:00:17.511867 (+ 827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:17.511871 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:17.511932 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:17.512035 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:17.513169 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.513662 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.514298 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.514472 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.517393 (+ 2921us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:17.517411 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:17.517414 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:17.517437 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:17.518958 (+ 1521us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.519444 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.519448 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.519450 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.519500 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.519814 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.519837 (+ 23us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.519839 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.520013 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:17.520110 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:17.520695 (+ 585us) server_negotiation.cc:300] Negotiation successful
0429 08:00:17.520788 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 08:00:17.521898 16185 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.510509 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37215 (local address 127.12.12.129:52553)
0429 08:00:17.510906 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:17.510937 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:17.511673 (+ 736us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:17.512185 (+ 512us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:17.512193 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:17.512557 (+ 364us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:17.513004 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:17.513014 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.513803 (+ 789us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.513806 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:17.514177 (+ 371us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:17.514182 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.514388 (+ 206us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.515432 (+ 1044us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:17.515449 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:17.517246 (+ 1797us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:17.519094 (+ 1848us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:17.519099 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:17.519110 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:17.519339 (+ 229us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:17.519592 (+ 253us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:17.519595 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:17.519597 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:17.519684 (+ 87us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:17.520156 (+ 472us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:17.520164 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:17.520422 (+ 258us) client_negotiation.cc:770] Sending connection context
0429 08:00:17.520665 (+ 243us) client_negotiation.cc:241] Negotiation successful
0429 08:00:17.520946 (+ 281us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":93,"threads_started":1}
I20260429 08:00:17.523108 16183 heartbeater.cc:344] Connected to a master server at 127.12.12.190:37215
I20260429 08:00:17.523394 16183 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:17.524019 16183 heartbeater.cc:507] Master 127.12.12.190:37215 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:00:17.525555 15990 ts_manager.cc:194] Registered new tserver with Master: 6211273f1790496394d30daa7151485f (127.12.12.129:36879)
I20260429 08:00:17.526692 15990 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:52553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:17.565500 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:37215
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:17.663450 16190 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:17.663679 16190 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:17.663800 16190 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:17.667104 16190 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:17.667160 16190 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:17.667315 16190 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:17.671455 16190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:17.672504 16190 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:17.673377 16190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 08:00:17.679775 16190 server_base.cc:1061] running on GCE node
W20260429 08:00:17.679636 16195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:17.679651 16198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:17.679636 16196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:17.680349 16190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:17.680959 16190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:17.682142 16190 hybrid_clock.cc:648] HybridClock initialized: now 1777449617682119 us; error 34 us; skew 500 ppm
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:17.685314 16190 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:00:17.686378 16190 webserver.cc:492] Webserver started at http://127.12.12.130:34407/ using document root <none> and password file <none>
I20260429 08:00:17.686965 16190 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:17.687039 16190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:17.687280 16190 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:17.688995 16190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "9474b34922d64800a26e2cb3f691d4db"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "4b5f588b1e14c969639bdb5d37314ce4"
server_key_iv: "93d5797ecb1589c9f5b2d98da5c9371d"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.689504 16190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "9474b34922d64800a26e2cb3f691d4db"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "4b5f588b1e14c969639bdb5d37314ce4"
server_key_iv: "93d5797ecb1589c9f5b2d98da5c9371d"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.692629 16190 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260429 08:00:17.694835 16205 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.695796 16190 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:17.695920 16190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "9474b34922d64800a26e2cb3f691d4db"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "4b5f588b1e14c969639bdb5d37314ce4"
server_key_iv: "93d5797ecb1589c9f5b2d98da5c9371d"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.696015 16190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:17.725589 16190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:17.728094 16190 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:17.728309 16190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:17.728924 16190 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:17.729818 16190 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:17.729889 16190 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.729960 16190 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:17.730013 16190 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.738896 16190 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:40935
I20260429 08:00:17.738937 16318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:40935 every 8 connection(s)
I20260429 08:00:17.739836 16190 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:17.741185 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16190
I20260429 08:00:17.741333 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:17.741601 12338 external_mini_cluster.cc:1468] Setting key 617572a1343ee34349b1f1771d1b66ce
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:00:17.752774 16029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.741768 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:42207 (local address 127.12.12.190:37215)
0429 08:00:17.741936 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:17.741941 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:17.742655 (+ 714us) server_negotiation.cc:408] Connection header received
0429 08:00:17.743511 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:17.743517 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:17.743577 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:17.743673 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:17.744845 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.745407 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.746037 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.746218 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.749251 (+ 3033us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:17.749278 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:17.749282 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:17.749312 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:17.750957 (+ 1645us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.751423 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.751426 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.751427 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.751471 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.751850 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.751852 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.751853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.751997 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:17.752075 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:17.752552 (+ 477us) server_negotiation.cc:300] Negotiation successful
0429 08:00:17.752650 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260429 08:00:17.753559 16321 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.742072 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37215 (local address 127.12.12.130:42207)
0429 08:00:17.742500 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:17.742551 (+ 51us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:17.743309 (+ 758us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:17.743813 (+ 504us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:17.743820 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:17.744208 (+ 388us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:17.744663 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:17.744673 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.745530 (+ 857us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.745536 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:17.745916 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:17.745922 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.746135 (+ 213us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.747241 (+ 1106us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:17.747258 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:17.749080 (+ 1822us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:17.751068 (+ 1988us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:17.751074 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:17.751085 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:17.751329 (+ 244us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:17.751635 (+ 306us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:17.751641 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:17.751646 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:17.751768 (+ 122us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:17.752086 (+ 318us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:17.752090 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:17.752312 (+ 222us) client_negotiation.cc:770] Sending connection context
0429 08:00:17.752521 (+ 209us) client_negotiation.cc:241] Negotiation successful
0429 08:00:17.752737 (+ 216us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:00:17.754737 16319 heartbeater.cc:344] Connected to a master server at 127.12.12.190:37215
I20260429 08:00:17.755009 16319 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:17.755658 16319 heartbeater.cc:507] Master 127.12.12.190:37215 requested a full tablet report, sending...
I20260429 08:00:17.756789 15990 ts_manager.cc:194] Registered new tserver with Master: 9474b34922d64800a26e2cb3f691d4db (127.12.12.130:40935)
I20260429 08:00:17.757647 15990 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:42207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:17.792865 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:37215
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:17.893573 16326 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:17.893793 16326 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:17.893894 16326 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:17.897195 16326 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:17.897253 16326 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:17.897378 16326 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:17.901820 16326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:17.902834 16326 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:17.903669 16326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:17.910051 16334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:17.910099 16326 server_base.cc:1061] running on GCE node
W20260429 08:00:17.910060 16332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:17.910054 16331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:17.910687 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:17.911239 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:17.912396 16326 hybrid_clock.cc:648] HybridClock initialized: now 1777449617912381 us; error 31 us; skew 500 ppm
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:17.915411 16326 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:00:17.916340 16326 webserver.cc:492] Webserver started at http://127.12.12.131:37505/ using document root <none> and password file <none>
I20260429 08:00:17.916890 16326 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:17.916937 16326 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:17.917099 16326 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:17.918815 16326 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "5fff1b49b2144822af3c34e1ba013c62"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "af21502c62a7210a6f72d42ed04b9351"
server_key_iv: "799f55cf4bfd3c1fac99f86d68ad0d37"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.919257 16326 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "5fff1b49b2144822af3c34e1ba013c62"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "af21502c62a7210a6f72d42ed04b9351"
server_key_iv: "799f55cf4bfd3c1fac99f86d68ad0d37"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.922350 16326 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:17.924374 16341 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.925364 16326 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:17.925504 16326 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "5fff1b49b2144822af3c34e1ba013c62"
format_stamp: "Formatted at 2026-04-29 08:00:17 on dist-test-slave-nhqz"
server_key: "af21502c62a7210a6f72d42ed04b9351"
server_key_iv: "799f55cf4bfd3c1fac99f86d68ad0d37"
server_key_version: "encryptionkey@0"
I20260429 08:00:17.925601 16326 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:17.934929 16326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:17.937486 16326 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:17.937719 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:17.938289 16326 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:17.939136 16326 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:17.939206 16326 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.939276 16326 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:17.939327 16326 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:17.948501 16326 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:39903
I20260429 08:00:17.948521 16454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:39903 every 8 connection(s)
I20260429 08:00:17.949568 16326 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:17.958244 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16326
I20260429 08:00:17.958384 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:17.958679 12338 external_mini_cluster.cc:1468] Setting key 850b7a06488d0b204558fe04fa61b97b
I20260429 08:00:17.961627 16029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.951357 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:51925 (local address 127.12.12.190:37215)
0429 08:00:17.951502 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:17.951505 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:17.952166 (+ 661us) server_negotiation.cc:408] Connection header received
0429 08:00:17.952976 (+ 810us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:17.952979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:17.953029 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:17.953105 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:17.954157 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.954627 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.955188 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.955306 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.957827 (+ 2521us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:17.957845 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:17.957847 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:17.957873 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:17.959544 (+ 1671us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.960170 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.960174 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.960176 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.960232 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.960566 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.960571 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.960573 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.960737 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:17.960883 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:17.961401 (+ 518us) server_negotiation.cc:300] Negotiation successful
0429 08:00:17.961507 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 08:00:17.962293 16457 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.951620 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37215 (local address 127.12.12.131:51925)
0429 08:00:17.952014 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:17.952046 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:17.952745 (+ 699us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:17.953249 (+ 504us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:17.953257 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:17.953606 (+ 349us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:17.954032 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:17.954042 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.954723 (+ 681us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.954726 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:17.955089 (+ 363us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:17.955095 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.955243 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.956197 (+ 954us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:17.956212 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:17.957682 (+ 1470us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:17.959673 (+ 1991us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:17.959678 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:17.959691 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:17.960046 (+ 355us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:17.960360 (+ 314us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:17.960365 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:17.960367 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:17.960466 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:17.960899 (+ 433us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:17.960905 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:17.961162 (+ 257us) client_negotiation.cc:770] Sending connection context
0429 08:00:17.961352 (+ 190us) client_negotiation.cc:241] Negotiation successful
0429 08:00:17.961548 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260429 08:00:17.963261 16455 heartbeater.cc:344] Connected to a master server at 127.12.12.190:37215
I20260429 08:00:17.963507 16455 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:17.964000 16455 heartbeater.cc:507] Master 127.12.12.190:37215 requested a full tablet report, sending...
I20260429 08:00:17.965015 15990 ts_manager.cc:194] Registered new tserver with Master: 5fff1b49b2144822af3c34e1ba013c62 (127.12.12.131:39903)
I20260429 08:00:17.965559 15990 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:51925
I20260429 08:00:17.972694 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Apr 29 08:00:17 dist-test-slave-nhqz krb5kdc[15942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449617, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:17.996227 16029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:17.988734 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.12.12.190:37215)
0429 08:00:17.988904 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:17.988908 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:17.989053 (+ 145us) server_negotiation.cc:408] Connection header received
0429 08:00:17.989166 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:17.989169 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:17.989218 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:17.989300 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:17.990097 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.990583 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:17.991257 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:17.991495 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:17.993627 (+ 2132us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:17.993645 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:17.993647 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:17.993673 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:17.994957 (+ 1284us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.995328 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.995331 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.995333 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.995381 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:17.995565 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:17.995567 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:17.995568 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:17.995724 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:17.995835 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:17.996014 (+ 179us) server_negotiation.cc:300] Negotiation successful
0429 08:00:17.996102 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260429 08:00:17.996646 15990 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41694
I20260429 08:00:17.997529 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16054
I20260429 08:00:18.003814 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16190
I20260429 08:00:18.009567 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16326
I20260429 08:00:18.015089 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 15958
2026-04-29T08:00:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3077 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:18 dist-test-slave-nhqz krb5kdc[16467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:18 dist-test-slave-nhqz krb5kdc[16467](info): set up 2 sockets
Apr 29 08:00:18 dist-test-slave-nhqz krb5kdc[16467](info): commencing operation
krb5kdc: starting...
W20260429 08:00:20.013857 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:20.155422 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:44317
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:44317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:20.258018 16483 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:20.258245 16483 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:20.258299 16483 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:20.261732 16483 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:20.261794 16483 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:20.261817 16483 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:20.261837 16483 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:20.261854 16483 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:20.266363 16483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:44317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:20.267633 16483 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:20.268620 16483 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:20.274482 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:20.274485 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:20.274593 16483 server_base.cc:1061] running on GCE node
W20260429 08:00:20.274482 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:20.275214 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:20.276150 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:20.277366 16483 hybrid_clock.cc:648] HybridClock initialized: now 1777449620277358 us; error 35 us; skew 500 ppm
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:20.280108 16483 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:20.281373 16483 webserver.cc:492] Webserver started at http://127.12.12.190:34193/ using document root <none> and password file <none>
I20260429 08:00:20.281934 16483 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:20.282014 16483 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:20.282248 16483 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:20.284710 16483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "401832b0e64743e2890cdac150919ca2"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "661f6a592aa257566ed19664e11a72a7"
server_key_iv: "0623b20941f76e49b56398970105ae57"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.285420 16483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "401832b0e64743e2890cdac150919ca2"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "661f6a592aa257566ed19664e11a72a7"
server_key_iv: "0623b20941f76e49b56398970105ae57"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.288985 16483 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:00:20.291275 16498 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.292388 16483 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:20.292495 16483 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "401832b0e64743e2890cdac150919ca2"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "661f6a592aa257566ed19664e11a72a7"
server_key_iv: "0623b20941f76e49b56398970105ae57"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.292584 16483 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:20.324658 16483 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:20.327659 16483 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:20.327824 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:20.335638 16550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:44317 every 8 connection(s)
I20260429 08:00:20.335640 16483 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:44317
I20260429 08:00:20.336741 16483 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:20.339596 16551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:20.342355 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16483
I20260429 08:00:20.342497 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:20.342727 12338 external_mini_cluster.cc:1468] Setting key 4c35407300887d7c44fbbc4ecb30588d
I20260429 08:00:20.345803 16551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2: Bootstrap starting.
I20260429 08:00:20.348313 16551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:20.349140 16551 log.cc:826] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2: Log is configured to *not* fsync() on all Append() calls
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:20.351107 16551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2: No bootstrap required, opened a new log
I20260429 08:00:20.353513 16551 raft_consensus.cc:359] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401832b0e64743e2890cdac150919ca2" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44317 } }
I20260429 08:00:20.353715 16551 raft_consensus.cc:385] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:20.353790 16551 raft_consensus.cc:740] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401832b0e64743e2890cdac150919ca2, State: Initialized, Role: FOLLOWER
I20260429 08:00:20.354231 16551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401832b0e64743e2890cdac150919ca2" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44317 } }
I20260429 08:00:20.354351 16551 raft_consensus.cc:399] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:20.354436 16551 raft_consensus.cc:493] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:20.354544 16551 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:20.355481 16551 raft_consensus.cc:515] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401832b0e64743e2890cdac150919ca2" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44317 } }
I20260429 08:00:20.355814 16551 leader_election.cc:304] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401832b0e64743e2890cdac150919ca2; no voters:
I20260429 08:00:20.356120 16551 leader_election.cc:290] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:20.356209 16556 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:20.356271 16554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:20.344071 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.12.12.190:44317)
0429 08:00:20.344544 (+ 473us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:20.344557 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:20.344598 (+ 41us) server_negotiation.cc:408] Connection header received
0429 08:00:20.345310 (+ 712us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:20.345333 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:20.345698 (+ 365us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:20.346042 (+ 344us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:20.346896 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.348054 (+ 1158us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:20.348752 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.349060 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:20.351309 (+ 2249us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:20.351327 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:20.351338 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:20.351366 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:20.353197 (+ 1831us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:20.353643 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:20.353648 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:20.353652 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:20.353738 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:20.354042 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:20.354045 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:20.354047 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:20.354434 (+ 387us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:20.354649 (+ 215us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:20.355059 (+ 410us) server_negotiation.cc:300] Negotiation successful
0429 08:00:20.355341 (+ 282us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":123,"threads_started":1}
I20260429 08:00:20.356408 16556 raft_consensus.cc:697] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [term 1 LEADER]: Becoming Leader. State: Replica: 401832b0e64743e2890cdac150919ca2, State: Running, Role: LEADER
I20260429 08:00:20.356699 16556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401832b0e64743e2890cdac150919ca2" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44317 } }
I20260429 08:00:20.357307 16551 sys_catalog.cc:565] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:20.358603 16558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401832b0e64743e2890cdac150919ca2. Latest consensus state: current_term: 1 leader_uuid: "401832b0e64743e2890cdac150919ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401832b0e64743e2890cdac150919ca2" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44317 } } }
I20260429 08:00:20.359011 16558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:20.358911 16557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401832b0e64743e2890cdac150919ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401832b0e64743e2890cdac150919ca2" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44317 } } }
I20260429 08:00:20.359341 16557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:20.359441 16569 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:20.361931 16569 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:20.367242 16569 catalog_manager.cc:1357] Generated new cluster ID: 4544bc3b0e2a4cf5ae395b5b8848c6c9
I20260429 08:00:20.367328 16569 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:20.378584 16569 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:20.379362 16569 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:20.388164 16569 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 401832b0e64743e2890cdac150919ca2: Generated new TSK 0
I20260429 08:00:20.388852 16569 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:20.440052 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:44317
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:20.539608 16579 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:20.539877 16579 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:20.539969 16579 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:20.543258 16579 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:20.543358 16579 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:20.543480 16579 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:20.547610 16579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:44317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:20.548739 16579 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:20.549589 16579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:20.555974 16587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:20.556074 16584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:20.556214 16579 server_base.cc:1061] running on GCE node
W20260429 08:00:20.555987 16585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:20.556581 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:20.557200 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:20.558374 16579 hybrid_clock.cc:648] HybridClock initialized: now 1777449620558355 us; error 43 us; skew 500 ppm
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:20.560966 16579 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:00:20.561946 16579 webserver.cc:492] Webserver started at http://127.12.12.129:43577/ using document root <none> and password file <none>
I20260429 08:00:20.562489 16579 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:20.562565 16579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:20.562776 16579 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:20.564416 16579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "ba0c07d2c2eb4625aab1aceb7a1a4fa4"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "36702de110e8df90ce83253965a3d569"
server_key_iv: "1bfb9909d07a05d0c54435b325a70a4c"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.564951 16579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "ba0c07d2c2eb4625aab1aceb7a1a4fa4"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "36702de110e8df90ce83253965a3d569"
server_key_iv: "1bfb9909d07a05d0c54435b325a70a4c"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.568230 16579 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 08:00:20.570322 16594 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.571213 16579 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:20.571339 16579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "ba0c07d2c2eb4625aab1aceb7a1a4fa4"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "36702de110e8df90ce83253965a3d569"
server_key_iv: "1bfb9909d07a05d0c54435b325a70a4c"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.571441 16579 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:20.583328 16579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:20.585806 16579 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:20.585994 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:20.586685 16579 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:20.587568 16579 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:20.587638 16579 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.587710 16579 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:20.587761 16579 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.596692 16579 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:44905
I20260429 08:00:20.596755 16707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:44905 every 8 connection(s)
I20260429 08:00:20.597771 16579 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:20.605994 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16579
I20260429 08:00:20.606128 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:20.606412 12338 external_mini_cluster.cc:1468] Setting key 1c5a07cb3ac2f5bae4a90f134f89ff43
I20260429 08:00:20.610157 16554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:20.599526 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:55389 (local address 127.12.12.190:44317)
0429 08:00:20.599676 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:20.599679 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:20.600298 (+ 619us) server_negotiation.cc:408] Connection header received
0429 08:00:20.601145 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:20.601148 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:20.601204 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:20.601310 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:20.602824 (+ 1514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.603348 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:20.604008 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.604199 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:20.606616 (+ 2417us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:20.606639 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:20.606646 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:20.606672 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:20.608171 (+ 1499us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:20.608691 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:20.608707 (+ 16us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:20.608708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:20.608752 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:20.609075 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:20.609079 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:20.609081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:20.609325 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:20.609436 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:20.609933 (+ 497us) server_negotiation.cc:300] Negotiation successful
0429 08:00:20.610033 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 08:00:20.610791 16710 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:20.599779 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:44317 (local address 127.12.12.129:55389)
0429 08:00:20.600164 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:20.600196 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:20.600954 (+ 758us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:20.601462 (+ 508us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:20.601470 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:20.601841 (+ 371us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:20.602669 (+ 828us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:20.602681 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.603484 (+ 803us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:20.603488 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:20.603885 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:20.603891 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.604113 (+ 222us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:20.604738 (+ 625us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:20.604755 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:20.606457 (+ 1702us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:20.608301 (+ 1844us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:20.608308 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:20.608320 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:20.608572 (+ 252us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:20.608857 (+ 285us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:20.608862 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:20.608864 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:20.608970 (+ 106us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:20.609450 (+ 480us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:20.609455 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:20.609669 (+ 214us) client_negotiation.cc:770] Sending connection context
0429 08:00:20.609847 (+ 178us) client_negotiation.cc:241] Negotiation successful
0429 08:00:20.610030 (+ 183us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260429 08:00:20.611961 16708 heartbeater.cc:344] Connected to a master server at 127.12.12.190:44317
I20260429 08:00:20.612224 16708 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:20.612916 16708 heartbeater.cc:507] Master 127.12.12.190:44317 requested a full tablet report, sending...
I20260429 08:00:20.614634 16515 ts_manager.cc:194] Registered new tserver with Master: ba0c07d2c2eb4625aab1aceb7a1a4fa4 (127.12.12.129:44905)
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:00:20.616286 16515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:55389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:20.655929 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:44317
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:20.755540 16715 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:20.755805 16715 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:20.755911 16715 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:20.759195 16715 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:20.759282 16715 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:20.759394 16715 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:20.763605 16715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:44317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:20.764679 16715 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:20.765512 16715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:20.771761 16723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:20.771790 16720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:20.771798 16721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:20.772006 16715 server_base.cc:1061] running on GCE node
I20260429 08:00:20.772364 16715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:20.772917 16715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:20.774098 16715 hybrid_clock.cc:648] HybridClock initialized: now 1777449620774088 us; error 32 us; skew 500 ppm
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:20.776921 16715 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:00:20.777978 16715 webserver.cc:492] Webserver started at http://127.12.12.130:37351/ using document root <none> and password file <none>
I20260429 08:00:20.778537 16715 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:20.778613 16715 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:20.778820 16715 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:20.780485 16715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "f5987385bf044a4cac5801380ee7a74d"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "cd180e8d11e29a7ae9b82c6ce9e343d7"
server_key_iv: "99b8d49315100a0c429b8e72d54569ce"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.781013 16715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "f5987385bf044a4cac5801380ee7a74d"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "cd180e8d11e29a7ae9b82c6ce9e343d7"
server_key_iv: "99b8d49315100a0c429b8e72d54569ce"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.784171 16715 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260429 08:00:20.786221 16730 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.787182 16715 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:20.787315 16715 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "f5987385bf044a4cac5801380ee7a74d"
format_stamp: "Formatted at 2026-04-29 08:00:20 on dist-test-slave-nhqz"
server_key: "cd180e8d11e29a7ae9b82c6ce9e343d7"
server_key_iv: "99b8d49315100a0c429b8e72d54569ce"
server_key_version: "encryptionkey@0"
I20260429 08:00:20.787413 16715 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:20.809001 16715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:20.811817 16715 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:20.812013 16715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:20.812568 16715 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:20.813485 16715 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:20.813557 16715 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.813627 16715 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:20.813687 16715 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:20.822755 16715 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:46879
I20260429 08:00:20.822791 16843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:46879 every 8 connection(s)
I20260429 08:00:20.823691 16715 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
Apr 29 08:00:20 dist-test-slave-nhqz krb5kdc[16467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449620, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:20.831904 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16715
I20260429 08:00:20.832026 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:20.832325 12338 external_mini_cluster.cc:1468] Setting key e73224a73bc8b050c3920646c3c969fd
I20260429 08:00:20.836062 16554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:20.825444 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41839 (local address 127.12.12.190:44317)
0429 08:00:20.825611 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:20.825615 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:20.826236 (+ 621us) server_negotiation.cc:408] Connection header received
0429 08:00:20.827008 (+ 772us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:20.827011 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:20.827062 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:20.827144 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:20.828738 (+ 1594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.829273 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:20.829962 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.830091 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:20.832321 (+ 2230us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:20.832341 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:20.832343 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:20.832391 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:20.833911 (+ 1520us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:20.834488 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:20.834491 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:20.834492 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:20.834533 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:20.834862 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:20.834864 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:20.834866 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:20.834997 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:20.835065 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:20.835716 (+ 651us) server_negotiation.cc:300] Negotiation successful
0429 08:00:20.835854 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 08:00:20.836599 16846 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:20.825700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:44317 (local address 127.12.12.130:41839)
0429 08:00:20.826087 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:20.826118 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:20.826835 (+ 717us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:20.827291 (+ 456us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:20.827299 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:20.827654 (+ 355us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:20.828599 (+ 945us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:20.828613 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.829375 (+ 762us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:20.829379 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:20.829845 (+ 466us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:20.829855 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:20.830026 (+ 171us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:20.830561 (+ 535us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:20.830578 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:20.832158 (+ 1580us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:20.834023 (+ 1865us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:20.834029 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:20.834042 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:20.834393 (+ 351us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:20.834623 (+ 230us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:20.834629 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:20.834634 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:20.834771 (+ 137us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:20.835092 (+ 321us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:20.835101 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:20.835405 (+ 304us) client_negotiation.cc:770] Sending connection context
0429 08:00:20.835650 (+ 245us) client_negotiation.cc:241] Negotiation successful
0429 08:00:20.835880 (+ 230us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260429 08:00:20.837785 16844 heartbeater.cc:344] Connected to a master server at 127.12.12.190:44317
I20260429 08:00:20.838073 16844 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:20.838557 16844 heartbeater.cc:507] Master 127.12.12.190:44317 requested a full tablet report, sending...
I20260429 08:00:20.839622 16515 ts_manager.cc:194] Registered new tserver with Master: f5987385bf044a4cac5801380ee7a74d (127.12.12.130:46879)
I20260429 08:00:20.840169 16515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:41839
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:20.881467 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:44317
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:20.982905 16851 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:20.983155 16851 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:20.983295 16851 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:20.986665 16851 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:20.986757 16851 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:20.986865 16851 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:20.991119 16851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:44317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:20.992274 16851 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:20.993180 16851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:20.999682 16859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:20.999686 16856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:21.000001 16851 server_base.cc:1061] running on GCE node
W20260429 08:00:20.999755 16857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:21.000492 16851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:21.001060 16851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:21.002218 16851 hybrid_clock.cc:648] HybridClock initialized: now 1777449621002199 us; error 32 us; skew 500 ppm
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:21.004886 16851 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:00:21.005915 16851 webserver.cc:492] Webserver started at http://127.12.12.131:45291/ using document root <none> and password file <none>
I20260429 08:00:21.006484 16851 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:21.006531 16851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:21.006716 16851 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:21.008369 16851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "ef56a1b3013e4bfe9011c06ec8127146"
format_stamp: "Formatted at 2026-04-29 08:00:21 on dist-test-slave-nhqz"
server_key: "7eb7a6b4dc60ac7c768e27aec0eb5cae"
server_key_iv: "3847b6cbcc4539a631b14a12e051a489"
server_key_version: "encryptionkey@0"
I20260429 08:00:21.008881 16851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "ef56a1b3013e4bfe9011c06ec8127146"
format_stamp: "Formatted at 2026-04-29 08:00:21 on dist-test-slave-nhqz"
server_key: "7eb7a6b4dc60ac7c768e27aec0eb5cae"
server_key_iv: "3847b6cbcc4539a631b14a12e051a489"
server_key_version: "encryptionkey@0"
I20260429 08:00:21.012233 16851 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 08:00:21.014298 16866 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:21.015239 16851 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 08:00:21.015362 16851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "ef56a1b3013e4bfe9011c06ec8127146"
format_stamp: "Formatted at 2026-04-29 08:00:21 on dist-test-slave-nhqz"
server_key: "7eb7a6b4dc60ac7c768e27aec0eb5cae"
server_key_iv: "3847b6cbcc4539a631b14a12e051a489"
server_key_version: "encryptionkey@0"
I20260429 08:00:21.015491 16851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:21.036849 16851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:21.039444 16851 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:21.039669 16851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:21.040238 16851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:21.041179 16851 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:21.041252 16851 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:21.041323 16851 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:21.041373 16851 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:21.050421 16851 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:39383
I20260429 08:00:21.050443 16979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:39383 every 8 connection(s)
I20260429 08:00:21.051340 16851 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:21.057413 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 16851
I20260429 08:00:21.057539 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:21.057837 12338 external_mini_cluster.cc:1468] Setting key 549d8c9ef64a86565ca40d84eac17684
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449621, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:21.063616 16554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:21.053178 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:55173 (local address 127.12.12.190:44317)
0429 08:00:21.053330 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:21.053334 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:21.053952 (+ 618us) server_negotiation.cc:408] Connection header received
0429 08:00:21.054715 (+ 763us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:21.054718 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:21.054766 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:21.054839 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:21.056266 (+ 1427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.056768 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:21.057569 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.057737 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:21.060237 (+ 2500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:21.060253 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:21.060256 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:21.060288 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:21.061760 (+ 1472us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:21.062255 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:21.062260 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:21.062262 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:21.062308 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:21.062592 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:21.062596 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:21.062598 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:21.062739 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:21.062842 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:21.063339 (+ 497us) server_negotiation.cc:300] Negotiation successful
0429 08:00:21.063460 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:00:21.064220 16982 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:21.053453 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:44317 (local address 127.12.12.131:55173)
0429 08:00:21.053818 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:21.053850 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:21.054548 (+ 698us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:21.054977 (+ 429us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:21.054984 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:21.055332 (+ 348us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:21.056145 (+ 813us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:21.056157 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.056902 (+ 745us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:21.056904 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:21.057433 (+ 529us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:21.057445 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.057643 (+ 198us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:21.058241 (+ 598us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:21.058258 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:21.060091 (+ 1833us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:21.061897 (+ 1806us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:21.061903 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:21.061915 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:21.062149 (+ 234us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:21.062400 (+ 251us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:21.062405 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:21.062412 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:21.062498 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:21.062841 (+ 343us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:21.062849 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:21.063101 (+ 252us) client_negotiation.cc:770] Sending connection context
0429 08:00:21.063304 (+ 203us) client_negotiation.cc:241] Negotiation successful
0429 08:00:21.063523 (+ 219us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260429 08:00:21.065233 16980 heartbeater.cc:344] Connected to a master server at 127.12.12.190:44317
I20260429 08:00:21.065490 16980 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:21.065980 16980 heartbeater.cc:507] Master 127.12.12.190:44317 requested a full tablet report, sending...
I20260429 08:00:21.066953 16515 ts_manager.cc:194] Registered new tserver with Master: ef56a1b3013e4bfe9011c06ec8127146 (127.12.12.131:39383)
I20260429 08:00:21.067514 16515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:55173
I20260429 08:00:21.071509 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 08:00:21.094913 16987 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:21.086885 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.12.12.129:44905)
0429 08:00:21.087160 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:21.087166 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:21.087188 (+ 22us) server_negotiation.cc:408] Connection header received
0429 08:00:21.087240 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:21.087245 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:21.087408 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:21.087529 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:21.088268 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.089017 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:21.089609 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.089802 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:21.091815 (+ 2013us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:21.091855 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:21.091867 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:21.091892 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:21.093617 (+ 1725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:21.093950 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:21.093954 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:21.093955 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:21.093996 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:21.094154 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:21.094155 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:21.094156 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:21.094300 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:21.094442 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:21.094568 (+ 126us) server_negotiation.cc:300] Negotiation successful
0429 08:00:21.094709 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:21.104151 16554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:21.097506 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.12.12.190:44317)
0429 08:00:21.097657 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:21.097661 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:21.097674 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:00:21.097764 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:21.097766 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:21.097808 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:21.097871 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:21.098476 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.098932 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:21.099484 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:21.099629 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:21.101708 (+ 2079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:21.101725 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:21.101728 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:21.101759 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:21.102926 (+ 1167us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:21.103261 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:21.103263 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:21.103264 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:21.103303 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:21.103491 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:21.103493 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:21.103494 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:21.103661 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:21.103759 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:21.103929 (+ 170us) server_negotiation.cc:300] Negotiation successful
0429 08:00:21.104018 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260429 08:00:21.104599 16515 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60124
I20260429 08:00:21.105259 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16579
I20260429 08:00:21.111605 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16715
I20260429 08:00:21.117117 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16851
I20260429 08:00:21.122793 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 16483
2026-04-29T08:00:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3108 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16992](info): set up 2 sockets
Apr 29 08:00:21 dist-test-slave-nhqz krb5kdc[16992](info): commencing operation
krb5kdc: starting...
W20260429 08:00:23.118083 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.973s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:23.254480 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:36639
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:36639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:23.357182 17008 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:23.357367 17008 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:23.357426 17008 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:23.360497 17008 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:23.360548 17008 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:23.360570 17008 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:23.360589 17008 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:23.360605 17008 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:23.364851 17008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:36639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:23.365831 17008 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:23.366621 17008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:23.371642 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:23.371654 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:23.371721 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:23.371924 17008 server_base.cc:1061] running on GCE node
I20260429 08:00:23.372320 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:23.373172 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:23.374330 17008 hybrid_clock.cc:648] HybridClock initialized: now 1777449623374306 us; error 41 us; skew 500 ppm
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:23.376758 17008 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:23.377507 17008 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:23.377592 17008 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:23.377806 17008 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:23.379505 17008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "1d0902e0f06c44d7a09bd4dd977c81a7"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "c0a715473be90a7e474ef2bd0eb094dd"
server_key_iv: "d0122b808ebd86de2e642a834c315486"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.380004 17008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "1d0902e0f06c44d7a09bd4dd977c81a7"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "c0a715473be90a7e474ef2bd0eb094dd"
server_key_iv: "d0122b808ebd86de2e642a834c315486"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.383196 17008 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260429 08:00:23.385188 17022 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.386103 17008 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:23.386261 17008 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "1d0902e0f06c44d7a09bd4dd977c81a7"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "c0a715473be90a7e474ef2bd0eb094dd"
server_key_iv: "d0122b808ebd86de2e642a834c315486"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.386359 17008 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:23.404074 17008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:23.410364 17008 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:23.410583 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:23.418049 17074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:36639 every 8 connection(s)
I20260429 08:00:23.418051 17008 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:36639
I20260429 08:00:23.418705 17008 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:23.420269 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17008
I20260429 08:00:23.420382 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:23.420634 12338 external_mini_cluster.cc:1468] Setting key ea8d3f6d11c320546d64d897249abef7
I20260429 08:00:23.422113 17075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:23.426889 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7: Bootstrap starting.
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:23.429347 17075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:23.430131 17075 log.cc:826] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:23.432077 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7: No bootstrap required, opened a new log
I20260429 08:00:23.433009 17078 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:23.422019 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.12.12.190:36639)
0429 08:00:23.422439 (+ 420us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:23.422448 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:23.422479 (+ 31us) server_negotiation.cc:408] Connection header received
0429 08:00:23.423095 (+ 616us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:23.423111 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:23.423362 (+ 251us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:23.423661 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:23.424553 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.425341 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:23.425996 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.426277 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:23.428629 (+ 2352us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:23.428664 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:23.428675 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:23.428701 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:23.430577 (+ 1876us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:23.431088 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:23.431093 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:23.431096 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:23.431179 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:23.431447 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:23.431450 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:23.431451 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:23.431726 (+ 275us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:23.431868 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:23.432199 (+ 331us) server_negotiation.cc:300] Negotiation successful
0429 08:00:23.432404 (+ 205us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260429 08:00:23.435207 17075 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 36639 } }
I20260429 08:00:23.435381 17075 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:23.435446 17075 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0902e0f06c44d7a09bd4dd977c81a7, State: Initialized, Role: FOLLOWER
I20260429 08:00:23.435792 17075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 36639 } }
I20260429 08:00:23.435918 17075 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:23.435976 17075 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:23.436081 17075 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:23.437167 17075 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 36639 } }
I20260429 08:00:23.437438 17075 leader_election.cc:304] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d0902e0f06c44d7a09bd4dd977c81a7; no voters:
I20260429 08:00:23.437657 17075 leader_election.cc:290] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:23.437788 17080 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:23.438033 17080 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0902e0f06c44d7a09bd4dd977c81a7, State: Running, Role: LEADER
I20260429 08:00:23.438436 17080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 36639 } }
I20260429 08:00:23.438843 17075 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:23.440168 17081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 36639 } } }
I20260429 08:00:23.440160 17082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d0902e0f06c44d7a09bd4dd977c81a7. Latest consensus state: current_term: 1 leader_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d0902e0f06c44d7a09bd4dd977c81a7" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 36639 } } }
I20260429 08:00:23.440284 17081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:23.440315 17082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:23.440665 17089 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:23.443440 17089 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:23.448302 17089 catalog_manager.cc:1357] Generated new cluster ID: 43a629f031744ba7aa51c150f47eb3f2
I20260429 08:00:23.448369 17089 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:23.476516 17089 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:23.477756 17089 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:23.495981 17089 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7: Generated new TSK 0
I20260429 08:00:23.496635 17089 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:23.555511 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:36639
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:23.657020 17103 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:23.657277 17103 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:23.657359 17103 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:23.660741 17103 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:23.660887 17103 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:23.661046 17103 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:23.665488 17103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:23.666636 17103 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:23.667544 17103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:23.673897 17111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:23.673972 17108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:23.673985 17109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:23.674067 17103 server_base.cc:1061] running on GCE node
I20260429 08:00:23.674503 17103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:23.675108 17103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:23.676285 17103 hybrid_clock.cc:648] HybridClock initialized: now 1777449623676275 us; error 33 us; skew 500 ppm
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:23.678965 17103 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:00:23.679698 17103 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:23.679781 17103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:23.679984 17103 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:23.681685 17103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "103c4cedcf374b789a9efad3fe502d2e"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "5ccc6cbafae689ca9c117aa3a0b57d3e"
server_key_iv: "a006245402dd9cdfeafae65c5872b33e"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.682202 17103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "103c4cedcf374b789a9efad3fe502d2e"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "5ccc6cbafae689ca9c117aa3a0b57d3e"
server_key_iv: "a006245402dd9cdfeafae65c5872b33e"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.685343 17103 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:00:23.687367 17117 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.688328 17103 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:23.688474 17103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "103c4cedcf374b789a9efad3fe502d2e"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "5ccc6cbafae689ca9c117aa3a0b57d3e"
server_key_iv: "a006245402dd9cdfeafae65c5872b33e"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.688578 17103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:23.702628 17103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:23.705000 17103 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:23.705188 17103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:23.705751 17103 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:23.706597 17103 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:23.706667 17103 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.706736 17103 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:23.706795 17103 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.716207 17103 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:42531
I20260429 08:00:23.716226 17230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:42531 every 8 connection(s)
I20260429 08:00:23.716926 17103 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:23.721688 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17103
I20260429 08:00:23.721817 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:23.722117 12338 external_mini_cluster.cc:1468] Setting key 76e64690d0cca3e0b63b50898a9f5714
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:23.729357 17078 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:23.718523 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:51837 (local address 127.12.12.190:36639)
0429 08:00:23.718657 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:23.718660 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:23.719234 (+ 574us) server_negotiation.cc:408] Connection header received
0429 08:00:23.720512 (+ 1278us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:23.720516 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:23.720576 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:23.720680 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:23.722012 (+ 1332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.722705 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:23.723465 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.723637 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:23.725794 (+ 2157us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:23.725821 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:23.725823 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:23.725846 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:23.727279 (+ 1433us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:23.727915 (+ 636us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:23.727918 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:23.727919 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:23.727958 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:23.728307 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:23.728309 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:23.728310 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:23.728443 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:23.728516 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:23.729142 (+ 626us) server_negotiation.cc:300] Negotiation successful
0429 08:00:23.729239 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 08:00:23.730238 17233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:23.718724 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:36639 (local address 127.12.12.129:51837)
0429 08:00:23.719104 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:23.719135 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:23.720309 (+ 1174us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:23.720883 (+ 574us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:23.720892 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:23.721225 (+ 333us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:23.721810 (+ 585us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:23.721827 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.722859 (+ 1032us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:23.722862 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:23.723333 (+ 471us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:23.723343 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.723560 (+ 217us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:23.724163 (+ 603us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:23.724181 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:23.725616 (+ 1435us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:23.727417 (+ 1801us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:23.727424 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:23.727438 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:23.727809 (+ 371us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:23.728053 (+ 244us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:23.728059 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:23.728062 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:23.728212 (+ 150us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:23.728552 (+ 340us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:23.728560 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:23.728867 (+ 307us) client_negotiation.cc:770] Sending connection context
0429 08:00:23.729122 (+ 255us) client_negotiation.cc:241] Negotiation successful
0429 08:00:23.729360 (+ 238us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:00:23.731426 17231 heartbeater.cc:344] Connected to a master server at 127.12.12.190:36639
I20260429 08:00:23.731657 17231 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:23.732116 17231 heartbeater.cc:507] Master 127.12.12.190:36639 requested a full tablet report, sending...
I20260429 08:00:23.733624 17039 ts_manager.cc:194] Registered new tserver with Master: 103c4cedcf374b789a9efad3fe502d2e (127.12.12.129:42531)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
I20260429 08:00:23.735153 17039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:51837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:23.771117 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:36639
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:23.872067 17238 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:23.872305 17238 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:23.872429 17238 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:23.876037 17238 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:23.876114 17238 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:23.876200 17238 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:23.880544 17238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:23.881677 17238 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:23.882483 17238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:23.888788 17243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:23.888788 17246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:23.888790 17244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:23.888864 17238 server_base.cc:1061] running on GCE node
I20260429 08:00:23.889411 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:23.889930 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:23.891079 17238 hybrid_clock.cc:648] HybridClock initialized: now 1777449623891049 us; error 45 us; skew 500 ppm
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:23.893821 17238 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:00:23.894614 17238 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:23.894701 17238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:23.894927 17238 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:23.897579 17238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "30b42037c1994a0b92a2871ade0eabc2"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "fe037e94cfbd475c320f60ad1dbcde8d"
server_key_iv: "8826678bae40d4a4a40a3d7e62a9b2bb"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.898211 17238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "30b42037c1994a0b92a2871ade0eabc2"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "fe037e94cfbd475c320f60ad1dbcde8d"
server_key_iv: "8826678bae40d4a4a40a3d7e62a9b2bb"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.903779 17238 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260429 08:00:23.906317 17252 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.907358 17238 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:23.907459 17238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "30b42037c1994a0b92a2871ade0eabc2"
format_stamp: "Formatted at 2026-04-29 08:00:23 on dist-test-slave-nhqz"
server_key: "fe037e94cfbd475c320f60ad1dbcde8d"
server_key_iv: "8826678bae40d4a4a40a3d7e62a9b2bb"
server_key_version: "encryptionkey@0"
I20260429 08:00:23.907544 17238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:23.919507 17238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:23.922089 17238 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:23.922251 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:23.922739 17238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:23.923595 17238 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:23.923642 17238 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.923681 17238 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:23.923696 17238 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:23.933360 17238 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:42333
I20260429 08:00:23.933391 17365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:42333 every 8 connection(s)
I20260429 08:00:23.933950 17238 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:23.937011 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17238
I20260429 08:00:23.937120 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:23.937428 12338 external_mini_cluster.cc:1468] Setting key d42954bee5976d7618254a873796f4a7
Apr 29 08:00:23 dist-test-slave-nhqz krb5kdc[16992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:00:23.947582 17078 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:23.935606 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:50717 (local address 127.12.12.190:36639)
0429 08:00:23.935767 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:23.935772 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:23.936408 (+ 636us) server_negotiation.cc:408] Connection header received
0429 08:00:23.938090 (+ 1682us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:23.938094 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:23.938166 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:23.938272 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:23.939708 (+ 1436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.940318 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:23.941264 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.941419 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:23.944058 (+ 2639us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:23.944074 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:23.944076 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:23.944104 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:23.945622 (+ 1518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:23.946133 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:23.946140 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:23.946142 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:23.946197 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:23.946527 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:23.946531 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:23.946533 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:23.946700 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:23.946806 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:23.947289 (+ 483us) server_negotiation.cc:300] Negotiation successful
0429 08:00:23.947418 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 08:00:23.948277 17368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:23.935875 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:36639 (local address 127.12.12.130:50717)
0429 08:00:23.936270 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:23.936301 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:23.937862 (+ 1561us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:23.938442 (+ 580us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:23.938451 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:23.938841 (+ 390us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:23.939506 (+ 665us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:23.939527 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.940449 (+ 922us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:23.940455 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:23.941149 (+ 694us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:23.941158 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:23.941317 (+ 159us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:23.941914 (+ 597us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:23.941932 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:23.943704 (+ 1772us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:23.945752 (+ 2048us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:23.945758 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:23.945771 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:23.946026 (+ 255us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:23.946306 (+ 280us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:23.946309 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:23.946310 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:23.946418 (+ 108us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:23.946807 (+ 389us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:23.946816 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:23.947042 (+ 226us) client_negotiation.cc:770] Sending connection context
0429 08:00:23.947256 (+ 214us) client_negotiation.cc:241] Negotiation successful
0429 08:00:23.947488 (+ 232us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:00:23.949595 17366 heartbeater.cc:344] Connected to a master server at 127.12.12.190:36639
I20260429 08:00:23.949864 17366 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:23.950382 17366 heartbeater.cc:507] Master 127.12.12.190:36639 requested a full tablet report, sending...
I20260429 08:00:23.951577 17039 ts_manager.cc:194] Registered new tserver with Master: 30b42037c1994a0b92a2871ade0eabc2 (127.12.12.130:42333)
I20260429 08:00:23.952149 17039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:50717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:23.985410 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:36639
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:24.088100 17373 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:24.088363 17373 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:24.088472 17373 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:24.091895 17373 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:24.092005 17373 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:24.092130 17373 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:24.096619 17373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:24.097800 17373 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:24.098785 17373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:24.105068 17378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.105068 17379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.105093 17381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:24.105458 17373 server_base.cc:1061] running on GCE node
I20260429 08:00:24.105815 17373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:24.106359 17373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:24.107540 17373 hybrid_clock.cc:648] HybridClock initialized: now 1777449624107519 us; error 35 us; skew 500 ppm
Apr 29 08:00:24 dist-test-slave-nhqz krb5kdc[16992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:24.110339 17373 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:00:24.111091 17373 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:24.111183 17373 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:24.111410 17373 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:24.113370 17373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "82d000a8c62647ad9cb2c052f1356ce1"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "8beda23f1a149d592b36364e5434cd52"
server_key_iv: "cf6b9168dd96cebd41fa2bf356373e7e"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.113852 17373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "82d000a8c62647ad9cb2c052f1356ce1"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "8beda23f1a149d592b36364e5434cd52"
server_key_iv: "cf6b9168dd96cebd41fa2bf356373e7e"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.117093 17373 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:00:24.119114 17387 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.120023 17373 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:24.120160 17373 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "82d000a8c62647ad9cb2c052f1356ce1"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "8beda23f1a149d592b36364e5434cd52"
server_key_iv: "cf6b9168dd96cebd41fa2bf356373e7e"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.120265 17373 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:24.139870 17373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:24.142565 17373 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:24.142711 17373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:24.143189 17373 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:24.143967 17373 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:24.144013 17373 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.144053 17373 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:24.144068 17373 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.152871 17500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:44351 every 8 connection(s)
I20260429 08:00:24.152896 17373 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:44351
I20260429 08:00:24.153452 17373 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
Apr 29 08:00:24 dist-test-slave-nhqz krb5kdc[16992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:24.161175 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17373
I20260429 08:00:24.161319 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:24.161613 12338 external_mini_cluster.cc:1468] Setting key a1c78815303eb773011c1c647e1ee778
I20260429 08:00:24.165330 17078 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.155081 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:47721 (local address 127.12.12.190:36639)
0429 08:00:24.155259 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.155263 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.155850 (+ 587us) server_negotiation.cc:408] Connection header received
0429 08:00:24.157111 (+ 1261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.157114 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.157171 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.157244 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:24.158269 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.158738 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.159310 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.159430 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.161585 (+ 2155us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:24.161605 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:24.161608 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:24.161636 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:24.163311 (+ 1675us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:24.163885 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:24.163889 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:24.163891 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:24.163939 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:24.164278 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:24.164282 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:24.164284 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:24.164426 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:24.164527 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.165031 (+ 504us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.165163 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260429 08:00:24.165863 17503 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.155333 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:36639 (local address 127.12.12.131:47721)
0429 08:00:24.155718 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.155749 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.156942 (+ 1193us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.157382 (+ 440us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.157390 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.157719 (+ 329us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:24.158137 (+ 418us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.158159 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.158831 (+ 672us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.158834 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:24.159217 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.159222 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.159362 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.159896 (+ 534us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:24.159913 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:24.161429 (+ 1516us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:24.163441 (+ 2012us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:24.163447 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:24.163458 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:24.163773 (+ 315us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:24.164049 (+ 276us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:24.164053 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:24.164056 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:24.164176 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:24.164527 (+ 351us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:24.164534 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:24.164746 (+ 212us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.164983 (+ 237us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.165197 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260429 08:00:24.166791 17501 heartbeater.cc:344] Connected to a master server at 127.12.12.190:36639
I20260429 08:00:24.167033 17501 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:24.167484 17501 heartbeater.cc:507] Master 127.12.12.190:36639 requested a full tablet report, sending...
I20260429 08:00:24.168417 17039 ts_manager.cc:194] Registered new tserver with Master: 82d000a8c62647ad9cb2c052f1356ce1 (127.12.12.131:44351)
I20260429 08:00:24.169041 17039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:47721
I20260429 08:00:24.175364 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 08:00:24.183090 17078 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.177114 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.12.12.190:36639)
0429 08:00:24.177239 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.177243 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.177360 (+ 117us) server_negotiation.cc:408] Connection header received
0429 08:00:24.177458 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.177460 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.177498 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.177561 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:24.178350 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.178814 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.179374 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.179494 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.180471 (+ 977us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:24.180484 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:24.180486 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:24.180508 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:24.181896 (+ 1388us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:24.182252 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:24.182254 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:24.182255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:24.182294 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:24.182477 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:24.182479 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:24.182481 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:24.182623 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:24.182719 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.182883 (+ 164us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.182977 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:24.185423 17039 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:24.187762 17039 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:24.200475 17512 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.196535 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:42333 (local address 127.0.0.1:56936)
0429 08:00:24.197130 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.197143 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.197222 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.197940 (+ 718us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.197943 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.197957 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:24.198243 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.198248 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.199226 (+ 978us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.199231 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:24.199987 (+ 756us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.199995 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.200130 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.200179 (+ 49us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.200263 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.200329 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"spinlock_wait_cycles":30592,"thread_start_us":83,"threads_started":1}
I20260429 08:00:24.200698 17513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.195854 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:44351 (local address 127.0.0.1:57138)
0429 08:00:24.196297 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.196330 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.196487 (+ 157us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.197407 (+ 920us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.197413 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.197435 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:24.197741 (+ 306us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.197749 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.199344 (+ 1595us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.199349 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:24.200336 (+ 987us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.200346 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.200442 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.200456 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.200498 (+ 42us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.200566 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":78,"threads_started":1}
I20260429 08:00:24.201218 17517 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.196939 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.12.12.130:42333)
0429 08:00:24.197399 (+ 460us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.197405 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.197423 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:00:24.197503 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.197507 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.197643 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.197798 (+ 155us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:24.198365 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.199114 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.200137 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.200806 (+ 669us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.200950 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.201031 (+ 81us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.201118 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":103,"threads_started":1}
I20260429 08:00:24.201515 17516 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.196584 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.12.12.129:42531)
0429 08:00:24.196961 (+ 377us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.196967 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.197303 (+ 336us) server_negotiation.cc:408] Connection header received
0429 08:00:24.197418 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.197423 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.197577 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.197706 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:24.198184 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.199282 (+ 1098us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.200645 (+ 1363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.201168 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.201254 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.201308 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.201383 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":88,"threads_started":1}
I20260429 08:00:24.201746 17515 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.195907 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:42531 (local address 127.0.0.1:60664)
0429 08:00:24.197092 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.197105 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.197210 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.197727 (+ 517us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.197734 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.197760 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:24.198023 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.198034 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.199389 (+ 1355us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.199392 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:24.200336 (+ 944us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.200346 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.200443 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.200456 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.200507 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.200566 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":98,"threads_started":1}
I20260429 08:00:24.201938 17514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.195971 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.12.12.131:44351)
0429 08:00:24.196763 (+ 792us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.196769 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.196791 (+ 22us) server_negotiation.cc:408] Connection header received
0429 08:00:24.196915 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.196921 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.197096 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.197398 (+ 302us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:24.197889 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.199066 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.201100 (+ 2034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.201549 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.201636 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.201694 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.201767 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":142,"threads_started":1}
I20260429 08:00:24.204069 17435 tablet_service.cc:1511] Processing CreateTablet for tablet 9a2f1e9cf2f2486983f20e8f176de54f (DEFAULT_TABLE table=test-table [id=7f19b1492723412ca07308c017c6623a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:24.204115 17300 tablet_service.cc:1511] Processing CreateTablet for tablet 9a2f1e9cf2f2486983f20e8f176de54f (DEFAULT_TABLE table=test-table [id=7f19b1492723412ca07308c017c6623a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:24.204114 17165 tablet_service.cc:1511] Processing CreateTablet for tablet 9a2f1e9cf2f2486983f20e8f176de54f (DEFAULT_TABLE table=test-table [id=7f19b1492723412ca07308c017c6623a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:24.205014 17300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2f1e9cf2f2486983f20e8f176de54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:24.205014 17435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2f1e9cf2f2486983f20e8f176de54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:24.205014 17165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2f1e9cf2f2486983f20e8f176de54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:24.210091 17518 tablet_bootstrap.cc:492] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Bootstrap starting.
I20260429 08:00:24.211308 17519 tablet_bootstrap.cc:492] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: Bootstrap starting.
I20260429 08:00:24.211377 17520 tablet_bootstrap.cc:492] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Bootstrap starting.
I20260429 08:00:24.212329 17518 tablet_bootstrap.cc:654] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:24.213208 17519 tablet_bootstrap.cc:654] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:24.213367 17520 tablet_bootstrap.cc:654] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:24.213618 17518 log.cc:826] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:24.214012 17519 log.cc:826] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:24.214097 17520 log.cc:826] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:24.215649 17519 tablet_bootstrap.cc:492] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: No bootstrap required, opened a new log
I20260429 08:00:24.215736 17518 tablet_bootstrap.cc:492] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: No bootstrap required, opened a new log
I20260429 08:00:24.215736 17520 tablet_bootstrap.cc:492] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: No bootstrap required, opened a new log
I20260429 08:00:24.215878 17519 ts_tablet_manager.cc:1403] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260429 08:00:24.215904 17520 ts_tablet_manager.cc:1403] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260429 08:00:24.215904 17518 ts_tablet_manager.cc:1403] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 08:00:24.218756 17519 raft_consensus.cc:359] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.219014 17519 raft_consensus.cc:385] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:24.219118 17519 raft_consensus.cc:740] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c4cedcf374b789a9efad3fe502d2e, State: Initialized, Role: FOLLOWER
I20260429 08:00:24.219365 17518 raft_consensus.cc:359] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.219365 17520 raft_consensus.cc:359] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.219555 17520 raft_consensus.cc:385] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:24.219622 17520 raft_consensus.cc:740] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b42037c1994a0b92a2871ade0eabc2, State: Initialized, Role: FOLLOWER
I20260429 08:00:24.219658 17518 raft_consensus.cc:385] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:24.219590 17519 consensus_queue.cc:260] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.219798 17518 raft_consensus.cc:740] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d000a8c62647ad9cb2c052f1356ce1, State: Initialized, Role: FOLLOWER
I20260429 08:00:24.220036 17520 consensus_queue.cc:260] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.220166 17518 consensus_queue.cc:260] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.220521 17519 ts_tablet_manager.cc:1434] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 08:00:24.220841 17501 heartbeater.cc:499] Master 127.12.12.190:36639 was elected leader, sending a full tablet report...
I20260429 08:00:24.220861 17231 heartbeater.cc:499] Master 127.12.12.190:36639 was elected leader, sending a full tablet report...
I20260429 08:00:24.220850 17518 ts_tablet_manager.cc:1434] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260429 08:00:24.221141 17366 heartbeater.cc:499] Master 127.12.12.190:36639 was elected leader, sending a full tablet report...
I20260429 08:00:24.221846 17520 ts_tablet_manager.cc:1434] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260429 08:00:24.275269 17525 raft_consensus.cc:493] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:24.275429 17525 raft_consensus.cc:515] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.276516 17525 leader_election.cc:290] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d000a8c62647ad9cb2c052f1356ce1 (127.12.12.131:44351), 103c4cedcf374b789a9efad3fe502d2e (127.12.12.129:42531)
I20260429 08:00:24.280058 17368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.276713 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:44351 (local address 127.12.12.130:56445)
0429 08:00:24.276924 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.276939 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.277033 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.277396 (+ 363us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.277399 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.277414 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:24.277688 (+ 274us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.277695 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.278708 (+ 1013us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.278711 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:24.279663 (+ 952us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.279671 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.279796 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.279806 (+ 10us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.279858 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.279910 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260429 08:00:24.280069 17527 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.276740 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:42531 (local address 127.12.12.130:55133)
0429 08:00:24.277170 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.277180 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.277250 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.277507 (+ 257us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.277510 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.277525 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:24.277764 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.277771 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.278708 (+ 937us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.278711 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:24.279663 (+ 952us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:24.279671 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.279788 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.279806 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.279858 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.279910 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":151,"threads_started":1}
I20260429 08:00:24.280494 17516 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.276968 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:55133 (local address 127.12.12.129:42531)
0429 08:00:24.277139 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.277143 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.277250 (+ 107us) server_negotiation.cc:408] Connection header received
0429 08:00:24.277327 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.277334 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.277410 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.277487 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:24.277887 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.278590 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.279785 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.280248 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.280284 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.280331 (+ 47us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.280381 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:00:24.280494 17514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.276824 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:56445 (local address 127.12.12.131:44351)
0429 08:00:24.276954 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.276961 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.276979 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:00:24.277189 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.277192 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.277267 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.277361 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:24.277809 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.278590 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.279784 (+ 1194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.280259 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.280285 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.280331 (+ 46us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.280381 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 08:00:24.281095 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a2f1e9cf2f2486983f20e8f176de54f" candidate_uuid: "30b42037c1994a0b92a2871ade0eabc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4cedcf374b789a9efad3fe502d2e" is_pre_election: true
I20260429 08:00:24.281097 17455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a2f1e9cf2f2486983f20e8f176de54f" candidate_uuid: "30b42037c1994a0b92a2871ade0eabc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d000a8c62647ad9cb2c052f1356ce1" is_pre_election: true
I20260429 08:00:24.281364 17185 raft_consensus.cc:2468] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b42037c1994a0b92a2871ade0eabc2 in term 0.
I20260429 08:00:24.281376 17455 raft_consensus.cc:2468] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b42037c1994a0b92a2871ade0eabc2 in term 0.
I20260429 08:00:24.281816 17254 leader_election.cc:304] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103c4cedcf374b789a9efad3fe502d2e, 30b42037c1994a0b92a2871ade0eabc2; no voters:
I20260429 08:00:24.282083 17525 raft_consensus.cc:2804] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:24.282162 17525 raft_consensus.cc:493] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:24.282207 17525 raft_consensus.cc:3060] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:24.283160 17525 raft_consensus.cc:515] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.283505 17525 leader_election.cc:290] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [CANDIDATE]: Term 1 election: Requested vote from peers 82d000a8c62647ad9cb2c052f1356ce1 (127.12.12.131:44351), 103c4cedcf374b789a9efad3fe502d2e (127.12.12.129:42531)
I20260429 08:00:24.283814 17455 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a2f1e9cf2f2486983f20e8f176de54f" candidate_uuid: "30b42037c1994a0b92a2871ade0eabc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d000a8c62647ad9cb2c052f1356ce1"
I20260429 08:00:24.283814 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a2f1e9cf2f2486983f20e8f176de54f" candidate_uuid: "30b42037c1994a0b92a2871ade0eabc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103c4cedcf374b789a9efad3fe502d2e"
I20260429 08:00:24.283941 17185 raft_consensus.cc:3060] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:24.283933 17455 raft_consensus.cc:3060] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:24.285171 17185 raft_consensus.cc:2468] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b42037c1994a0b92a2871ade0eabc2 in term 1.
I20260429 08:00:24.285166 17455 raft_consensus.cc:2468] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b42037c1994a0b92a2871ade0eabc2 in term 1.
I20260429 08:00:24.285477 17255 leader_election.cc:304] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b42037c1994a0b92a2871ade0eabc2, 82d000a8c62647ad9cb2c052f1356ce1; no voters:
I20260429 08:00:24.285634 17525 raft_consensus.cc:2804] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:24.285845 17525 raft_consensus.cc:697] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 1 LEADER]: Becoming Leader. State: Replica: 30b42037c1994a0b92a2871ade0eabc2, State: Running, Role: LEADER
I20260429 08:00:24.286149 17525 consensus_queue.cc:237] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } }
I20260429 08:00:24.289299 17037 catalog_manager.cc:5671] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30b42037c1994a0b92a2871ade0eabc2 (127.12.12.130). New cstate: current_term: 1 leader_uuid: "30b42037c1994a0b92a2871ade0eabc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30b42037c1994a0b92a2871ade0eabc2" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 42333 } health_report { overall_health: HEALTHY } } }
I20260429 08:00:24.301425 17517 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.297712 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.12.12.130:42333)
0429 08:00:24.297853 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.297857 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.297871 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:00:24.298019 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.298022 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.298070 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.298180 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:24.298577 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.299264 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:24.300077 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:24.300309 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:24.300447 (+ 138us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:24.300991 (+ 544us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:24.301099 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.301225 (+ 126us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.301278 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 08:00:24.306891 17455 raft_consensus.cc:1275] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 1 FOLLOWER]: Refusing update from remote peer 30b42037c1994a0b92a2871ade0eabc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:24.306891 17185 raft_consensus.cc:1275] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 1 FOLLOWER]: Refusing update from remote peer 30b42037c1994a0b92a2871ade0eabc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:24.307574 17528 consensus_queue.cc:1048] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d000a8c62647ad9cb2c052f1356ce1" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:24.307752 17525 consensus_queue.cc:1048] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103c4cedcf374b789a9efad3fe502d2e" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:24.314996 17532 mvcc.cc:204] Tried to move back new op lower bound from 7280433661154070528 to 7280433661078900736. Current Snapshot: MvccSnapshot[applied={T|T < 7280433661154070528}]
I20260429 08:00:24.322731 17037 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:24.322929 17037 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:24.325232 17037 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1d0902e0f06c44d7a09bd4dd977c81a7: Sending DeleteTablet for 3 replicas of tablet 9a2f1e9cf2f2486983f20e8f176de54f
I20260429 08:00:24.326020 17435 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a2f1e9cf2f2486983f20e8f176de54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:24 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:57138
I20260429 08:00:24.326021 17300 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a2f1e9cf2f2486983f20e8f176de54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:24 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:56936
I20260429 08:00:24.326021 17165 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a2f1e9cf2f2486983f20e8f176de54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:24 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:60664
I20260429 08:00:24.326545 17543 tablet_replica.cc:333] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e: stopping tablet replica
I20260429 08:00:24.326818 17543 raft_consensus.cc:2243] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:24.326998 17542 tablet_replica.cc:333] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: stopping tablet replica
I20260429 08:00:24.327152 17543 raft_consensus.cc:2272] T 9a2f1e9cf2f2486983f20e8f176de54f P 103c4cedcf374b789a9efad3fe502d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:24.327232 17542 raft_consensus.cc:2243] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:24.327456 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17103
I20260429 08:00:24.328908 17542 raft_consensus.cc:2272] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:24.329017 17541 tablet_replica.cc:333] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: stopping tablet replica
I20260429 08:00:24.329263 17541 raft_consensus.cc:2243] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:24.329641 17541 raft_consensus.cc:2272] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:24.330384 17542 ts_tablet_manager.cc:1916] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:24.330861 17541 ts_tablet_manager.cc:1916] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:24.333012 17542 ts_tablet_manager.cc:1929] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:24.333109 17542 log.cc:1199] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/9a2f1e9cf2f2486983f20e8f176de54f
I20260429 08:00:24.333384 17541 ts_tablet_manager.cc:1929] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:24.333480 17541 log.cc:1199] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/9a2f1e9cf2f2486983f20e8f176de54f
I20260429 08:00:24.333480 17542 ts_tablet_manager.cc:1950] T 9a2f1e9cf2f2486983f20e8f176de54f P 82d000a8c62647ad9cb2c052f1356ce1: Deleting consensus metadata
I20260429 08:00:24.333770 17541 ts_tablet_manager.cc:1950] T 9a2f1e9cf2f2486983f20e8f176de54f P 30b42037c1994a0b92a2871ade0eabc2: Deleting consensus metadata
I20260429 08:00:24.334717 17025 catalog_manager.cc:5002] TS 82d000a8c62647ad9cb2c052f1356ce1 (127.12.12.131:44351): tablet 9a2f1e9cf2f2486983f20e8f176de54f (table test-table [id=7f19b1492723412ca07308c017c6623a]) successfully deleted
I20260429 08:00:24.334985 17025 catalog_manager.cc:5002] TS 30b42037c1994a0b92a2871ade0eabc2 (127.12.12.130:42333): tablet 9a2f1e9cf2f2486983f20e8f176de54f (table test-table [id=7f19b1492723412ca07308c017c6623a]) successfully deleted
W20260429 08:00:24.336669 17024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.129:42531 (error 108)
I20260429 08:00:24.337478 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17238
I20260429 08:00:24.338203 17515 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.337397 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:42531 (local address 127.0.0.1:60668)
0429 08:00:24.338073 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.338124 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567}
W20260429 08:00:24.339015 17024 catalog_manager.cc:4729] TS 103c4cedcf374b789a9efad3fe502d2e (127.12.12.129:42531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a2f1e9cf2f2486983f20e8f176de54f: Network error: Client connection negotiation failed: client connection to 127.12.12.129:42531: connect: Connection refused (error 111)
I20260429 08:00:24.343923 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17373
I20260429 08:00:24.349933 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17008
2026-04-29T08:00:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3226 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-04-29T08:00:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:24Z Disabled control of system clock
I20260429 08:00:24.379053 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:45809
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 08:00:24.480780 17548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:24.481069 17548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:24.481117 17548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:24.484395 17548 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:24.484453 17548 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:24.484470 17548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:24.484488 17548 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:24.484505 17548 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:24.489140 17548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:45809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:24.490211 17548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:24.491318 17548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:24.496840 17556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.496852 17554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.496924 17553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:24.497088 17548 server_base.cc:1061] running on GCE node
I20260429 08:00:24.497588 17548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:24.498422 17548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:24.499601 17548 hybrid_clock.cc:648] HybridClock initialized: now 1777449624499584 us; error 38 us; skew 500 ppm
I20260429 08:00:24.501335 17548 webserver.cc:492] Webserver started at http://127.12.12.190:32899/ using document root <none> and password file <none>
I20260429 08:00:24.501855 17548 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:24.501956 17548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:24.502174 17548 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:24.503775 17548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "67693de12ffb4be285be42fa16d10a21"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "024661376add9cb344a4e636c7a7576d"
server_key_iv: "b65e31f12a3c1d87ca0bc8fe094330ca"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.504271 17548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "67693de12ffb4be285be42fa16d10a21"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "024661376add9cb344a4e636c7a7576d"
server_key_iv: "b65e31f12a3c1d87ca0bc8fe094330ca"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.507493 17548 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:24.509641 17562 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.510556 17548 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:24.510697 17548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "67693de12ffb4be285be42fa16d10a21"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "024661376add9cb344a4e636c7a7576d"
server_key_iv: "b65e31f12a3c1d87ca0bc8fe094330ca"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.510794 17548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:24.515242 17548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:24.515763 17548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:24.515931 17548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:24.522424 17548 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:45809
I20260429 08:00:24.522436 17614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:45809 every 8 connection(s)
I20260429 08:00:24.523353 17548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:24.523707 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17548
I20260429 08:00:24.523847 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:24.524082 12338 external_mini_cluster.cc:1468] Setting key 286c4b1d40f7b6996e8ecc1ced8d7d47
I20260429 08:00:24.526263 17615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:24.528908 17618 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.525309 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.12.12.190:45809)
0429 08:00:24.525829 (+ 520us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.525837 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.525865 (+ 28us) server_negotiation.cc:408] Connection header received
0429 08:00:24.526542 (+ 677us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.526560 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.526820 (+ 260us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.527115 (+ 295us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:24.527897 (+ 782us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:24.527902 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:24.527928 (+ 26us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:24.527938 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:24.527969 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:24.527978 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:24.528135 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.528238 (+ 103us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.528359 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":119,"threads_started":1}
I20260429 08:00:24.531839 17615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21: Bootstrap starting.
I20260429 08:00:24.533804 17615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:24.534462 17615 log.cc:826] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:24.536175 17615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21: No bootstrap required, opened a new log
I20260429 08:00:24.538529 17615 raft_consensus.cc:359] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67693de12ffb4be285be42fa16d10a21" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45809 } }
I20260429 08:00:24.538710 17615 raft_consensus.cc:385] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:24.538793 17615 raft_consensus.cc:740] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67693de12ffb4be285be42fa16d10a21, State: Initialized, Role: FOLLOWER
I20260429 08:00:24.539225 17615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67693de12ffb4be285be42fa16d10a21" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45809 } }
I20260429 08:00:24.539367 17615 raft_consensus.cc:399] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:24.539430 17615 raft_consensus.cc:493] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:24.539517 17615 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:24.540309 17615 raft_consensus.cc:515] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67693de12ffb4be285be42fa16d10a21" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45809 } }
I20260429 08:00:24.540620 17615 leader_election.cc:304] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67693de12ffb4be285be42fa16d10a21; no voters:
I20260429 08:00:24.540923 17615 leader_election.cc:290] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:24.541040 17620 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:24.541272 17620 raft_consensus.cc:697] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [term 1 LEADER]: Becoming Leader. State: Replica: 67693de12ffb4be285be42fa16d10a21, State: Running, Role: LEADER
I20260429 08:00:24.541531 17620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67693de12ffb4be285be42fa16d10a21" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45809 } }
I20260429 08:00:24.542011 17615 sys_catalog.cc:565] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:24.543004 17621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67693de12ffb4be285be42fa16d10a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67693de12ffb4be285be42fa16d10a21" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45809 } } }
I20260429 08:00:24.543134 17621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:24.543478 17629 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:24.543457 17622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67693de12ffb4be285be42fa16d10a21. Latest consensus state: current_term: 1 leader_uuid: "67693de12ffb4be285be42fa16d10a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67693de12ffb4be285be42fa16d10a21" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45809 } } }
I20260429 08:00:24.544140 17622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:24.546582 17629 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:24.551215 17629 catalog_manager.cc:1357] Generated new cluster ID: 6f54f80725574cdea5c021043a04df07
I20260429 08:00:24.551290 17629 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:24.566749 17629 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:24.567046 17629 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:24.574841 17629 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21: Generated new TSK 0
I20260429 08:00:24.575382 17629 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:00:24.582741 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45809
--builtin_ntp_servers=127.12.12.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 08:00:24.684286 17639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:24.684500 17639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:24.684548 17639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:24.687834 17639 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:24.687886 17639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:24.687999 17639 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:24.691903 17639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:24.692950 17639 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:24.694018 17639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:24.700094 17644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.700120 17647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.700243 17645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:24.700425 17639 server_base.cc:1061] running on GCE node
I20260429 08:00:24.700765 17639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:24.701445 17639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:24.702637 17639 hybrid_clock.cc:648] HybridClock initialized: now 1777449624702617 us; error 35 us; skew 500 ppm
I20260429 08:00:24.704604 17639 webserver.cc:492] Webserver started at http://127.12.12.129:32859/ using document root <none> and password file <none>
I20260429 08:00:24.705271 17639 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:24.705346 17639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:24.705557 17639 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:24.707274 17639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "ac400030a5d34b03a981bdef14f105ce"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "96b723dbb63328a28053f2e8e59e1768"
server_key_iv: "8306b490a2ee8708577229cbe412fa84"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.707783 17639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "ac400030a5d34b03a981bdef14f105ce"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "96b723dbb63328a28053f2e8e59e1768"
server_key_iv: "8306b490a2ee8708577229cbe412fa84"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.710839 17639 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:24.712795 17653 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.713790 17639 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:24.713907 17639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "ac400030a5d34b03a981bdef14f105ce"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "96b723dbb63328a28053f2e8e59e1768"
server_key_iv: "8306b490a2ee8708577229cbe412fa84"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.714037 17639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:24.718732 17639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:24.719266 17639 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:24.719442 17639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:24.719919 17639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:24.720798 17639 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:24.720927 17639 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.720992 17639 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:24.721052 17639 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.729816 17639 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:39253
I20260429 08:00:24.729853 17766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:39253 every 8 connection(s)
I20260429 08:00:24.730696 17639 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:24.735672 17618 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.732380 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:57303 (local address 127.12.12.190:45809)
0429 08:00:24.732535 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.732539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.733384 (+ 845us) server_negotiation.cc:408] Connection header received
0429 08:00:24.734156 (+ 772us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.734159 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.734200 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.734282 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:24.735232 (+ 950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:24.735236 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:24.735238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:24.735251 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:24.735274 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:24.735312 (+ 38us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:24.735414 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.735525 (+ 111us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.735557 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260429 08:00:24.736228 17769 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.732619 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45809 (local address 127.12.12.129:57303)
0429 08:00:24.733231 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.733261 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.733935 (+ 674us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.734424 (+ 489us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.734432 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.734491 (+ 59us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:24.734992 (+ 501us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:24.734994 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:24.735016 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:24.735059 (+ 43us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:24.735399 (+ 340us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:24.735404 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:24.735408 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.735538 (+ 130us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.735616 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":90,"threads_started":1}
I20260429 08:00:24.737584 17767 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45809
I20260429 08:00:24.737785 17767 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:24.738130 17767 heartbeater.cc:507] Master 127.12.12.190:45809 requested a full tablet report, sending...
I20260429 08:00:24.738199 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17639
I20260429 08:00:24.738353 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:24.738626 12338 external_mini_cluster.cc:1468] Setting key bc9d09f19c190288aa79d8c2cfb43d42
I20260429 08:00:24.739745 17579 ts_manager.cc:194] Registered new tserver with Master: ac400030a5d34b03a981bdef14f105ce (127.12.12.129:39253)
I20260429 08:00:24.740712 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45809
--builtin_ntp_servers=127.12.12.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 08:00:24.842610 17770 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:24.842809 17770 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:24.842859 17770 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:24.846308 17770 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:24.846365 17770 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:24.846458 17770 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:24.850456 17770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:24.851495 17770 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:24.852559 17770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:24.858994 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.858994 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:24.859040 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:24.859382 17770 server_base.cc:1061] running on GCE node
I20260429 08:00:24.859746 17770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:24.860291 17770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:24.861451 17770 hybrid_clock.cc:648] HybridClock initialized: now 1777449624861440 us; error 28 us; skew 500 ppm
I20260429 08:00:24.863104 17770 webserver.cc:492] Webserver started at http://127.12.12.130:36485/ using document root <none> and password file <none>
I20260429 08:00:24.863659 17770 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:24.863739 17770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:24.863951 17770 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:24.865592 17770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "48404e8dbbfa4fd7820fa5374bdbbd59"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "1e1dde0705fd5f209c66ce176b82804f"
server_key_iv: "8770ff0e6a6500361f4513a5b8b305b8"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.866050 17770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "48404e8dbbfa4fd7820fa5374bdbbd59"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "1e1dde0705fd5f209c66ce176b82804f"
server_key_iv: "8770ff0e6a6500361f4513a5b8b305b8"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.869143 17770 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:00:24.871178 17784 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.872108 17770 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:24.872241 17770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "48404e8dbbfa4fd7820fa5374bdbbd59"
format_stamp: "Formatted at 2026-04-29 08:00:24 on dist-test-slave-nhqz"
server_key: "1e1dde0705fd5f209c66ce176b82804f"
server_key_iv: "8770ff0e6a6500361f4513a5b8b305b8"
server_key_version: "encryptionkey@0"
I20260429 08:00:24.872336 17770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:24.876487 17770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:24.877055 17770 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:24.877230 17770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:24.877704 17770 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:24.878566 17770 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:24.878633 17770 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.878695 17770 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:24.878738 17770 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:24.887490 17770 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:46321
I20260429 08:00:24.887554 17897 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:46321 every 8 connection(s)
I20260429 08:00:24.888414 17770 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:24.893496 17618 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.890099 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41083 (local address 127.12.12.190:45809)
0429 08:00:24.890250 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:24.890254 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:24.891003 (+ 749us) server_negotiation.cc:408] Connection header received
0429 08:00:24.891778 (+ 775us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:24.891781 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:24.891828 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:24.891932 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:24.893005 (+ 1073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:24.893009 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:24.893013 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:24.893024 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:24.893054 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:24.893063 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:24.893180 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:24.893324 (+ 144us) server_negotiation.cc:300] Negotiation successful
0429 08:00:24.893359 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 08:00:24.894025 17900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:24.890351 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45809 (local address 127.12.12.130:41083)
0429 08:00:24.890877 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:24.890905 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:24.891587 (+ 682us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:24.892107 (+ 520us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:24.892114 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:24.892173 (+ 59us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:24.892734 (+ 561us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:24.892736 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:24.892759 (+ 23us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:24.892849 (+ 90us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:24.893168 (+ 319us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:24.893174 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:24.893179 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 08:00:24.893364 (+ 185us) client_negotiation.cc:241] Negotiation successful
0429 08:00:24.893464 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260429 08:00:24.895193 17898 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45809
I20260429 08:00:24.895412 17898 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:24.895816 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17770
I20260429 08:00:24.895867 17898 heartbeater.cc:507] Master 127.12.12.190:45809 requested a full tablet report, sending...
I20260429 08:00:24.895931 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:24.896209 12338 external_mini_cluster.cc:1468] Setting key 3437f42d2fd7750ab64ce43d41a8aa65
I20260429 08:00:24.896970 17579 ts_manager.cc:194] Registered new tserver with Master: 48404e8dbbfa4fd7820fa5374bdbbd59 (127.12.12.130:46321)
I20260429 08:00:24.898047 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45809
--builtin_ntp_servers=127.12.12.148:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 08:00:24.999559 17901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:24.999752 17901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:24.999814 17901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:25.003012 17901 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:25.003060 17901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:25.003191 17901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:25.007009 17901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:25.008085 17901 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:25.009168 17901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:25.015444 17906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:25.015424 17907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:25.015637 17909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:25.015739 17901 server_base.cc:1061] running on GCE node
I20260429 08:00:25.016194 17901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:25.016731 17901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:25.017884 17901 hybrid_clock.cc:648] HybridClock initialized: now 1777449625017870 us; error 32 us; skew 500 ppm
I20260429 08:00:25.019645 17901 webserver.cc:492] Webserver started at http://127.12.12.131:34179/ using document root <none> and password file <none>
I20260429 08:00:25.020170 17901 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:25.020223 17901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:25.020459 17901 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:25.022137 17901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "5f6d9a97873148318e2328a5a7d432c2"
format_stamp: "Formatted at 2026-04-29 08:00:25 on dist-test-slave-nhqz"
server_key: "c9c9bb3e418faee24c7d13945bab52bf"
server_key_iv: "044365bd68ea49954692e36b676eb70c"
server_key_version: "encryptionkey@0"
I20260429 08:00:25.022704 17901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "5f6d9a97873148318e2328a5a7d432c2"
format_stamp: "Formatted at 2026-04-29 08:00:25 on dist-test-slave-nhqz"
server_key: "c9c9bb3e418faee24c7d13945bab52bf"
server_key_iv: "044365bd68ea49954692e36b676eb70c"
server_key_version: "encryptionkey@0"
I20260429 08:00:25.025894 17901 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260429 08:00:25.028018 17915 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:25.029076 17901 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:00:25.029197 17901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "5f6d9a97873148318e2328a5a7d432c2"
format_stamp: "Formatted at 2026-04-29 08:00:25 on dist-test-slave-nhqz"
server_key: "c9c9bb3e418faee24c7d13945bab52bf"
server_key_iv: "044365bd68ea49954692e36b676eb70c"
server_key_version: "encryptionkey@0"
I20260429 08:00:25.029299 17901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:25.033545 17901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:25.034047 17901 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:25.034228 17901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:25.034708 17901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:25.035633 17901 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:25.035704 17901 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:25.035768 17901 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:25.035812 17901 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:25.044257 17901 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:39633
I20260429 08:00:25.044340 18028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:39633 every 8 connection(s)
I20260429 08:00:25.045243 17901 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:25.050239 17618 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.046942 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:52125 (local address 127.12.12.190:45809)
0429 08:00:25.047076 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.047079 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.047907 (+ 828us) server_negotiation.cc:408] Connection header received
0429 08:00:25.048690 (+ 783us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.048694 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.048741 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.048887 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.049790 (+ 903us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.049794 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.049796 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.049808 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.049837 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.049845 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.049938 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.050072 (+ 134us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.050106 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 08:00:25.050751 18031 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.047244 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45809 (local address 127.12.12.131:52125)
0429 08:00:25.047776 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.047805 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.048507 (+ 702us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.049023 (+ 516us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.049030 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.049086 (+ 56us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:25.049592 (+ 506us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:25.049593 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:25.049615 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:25.049656 (+ 41us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:25.049924 (+ 268us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:25.049931 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:25.049936 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 08:00:25.050112 (+ 176us) client_negotiation.cc:241] Negotiation successful
0429 08:00:25.050214 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260429 08:00:25.051849 18029 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45809
I20260429 08:00:25.052081 18029 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:25.052443 18029 heartbeater.cc:507] Master 127.12.12.190:45809 requested a full tablet report, sending...
I20260429 08:00:25.053306 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 17901
I20260429 08:00:25.053416 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:25.053510 17579 ts_manager.cc:194] Registered new tserver with Master: 5f6d9a97873148318e2328a5a7d432c2 (127.12.12.131:39633)
I20260429 08:00:25.053678 12338 external_mini_cluster.cc:1468] Setting key e3e391146ba584c8665739be71817895
I20260429 08:00:25.055487 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 08:00:25.058912 17618 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.057199 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.12.12.190:45809)
0429 08:00:25.057343 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.057347 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.057500 (+ 153us) server_negotiation.cc:408] Connection header received
0429 08:00:25.057646 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.057649 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.057683 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.057795 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.058521 (+ 726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.058524 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.058526 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.058535 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.058552 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.058559 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.058641 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.058717 (+ 76us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.058741 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 08:00:25.061218 17579 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:25.063580 17579 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:25.074025 18037 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.070939 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46321 (local address 127.0.0.1:50976)
0429 08:00:25.071619 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.071651 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.071749 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.072494 (+ 745us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.072500 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.072556 (+ 56us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:25.073280 (+ 724us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:25.073282 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:25.073300 (+ 18us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:25.073354 (+ 54us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:25.073693 (+ 339us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:25.073700 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:25.073705 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 08:00:25.073823 (+ 118us) client_negotiation.cc:241] Negotiation successful
0429 08:00:25.073865 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":101,"threads_started":1}
I20260429 08:00:25.074394 18038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.071125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.12.12.130:46321)
0429 08:00:25.071494 (+ 369us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.071501 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.071850 (+ 349us) server_negotiation.cc:408] Connection header received
0429 08:00:25.071910 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.071915 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.072093 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.072246 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.073505 (+ 1259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.073512 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.073521 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.073531 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.073571 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.073584 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.074015 (+ 431us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.074063 (+ 48us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.074232 (+ 169us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
I20260429 08:00:25.074453 18042 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.071952 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.12.12.129:39253)
0429 08:00:25.072378 (+ 426us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.072383 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.072404 (+ 21us) server_negotiation.cc:408] Connection header received
0429 08:00:25.072540 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.072545 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.072693 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.072846 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.073666 (+ 820us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.073674 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.073683 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.073694 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.073728 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.073759 (+ 31us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.073883 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.073999 (+ 116us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.074063 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260429 08:00:25.074738 18039 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.071220 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:39633 (local address 127.0.0.1:34012)
0429 08:00:25.072246 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.072260 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.072328 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.073503 (+ 1175us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.073506 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.073521 (+ 15us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:25.074088 (+ 567us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:25.074089 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:25.074101 (+ 12us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:25.074124 (+ 23us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:25.074474 (+ 350us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:25.074481 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:25.074487 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 08:00:25.074586 (+ 99us) client_negotiation.cc:241] Negotiation successful
0429 08:00:25.074611 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":77,"threads_started":1}
I20260429 08:00:25.074935 18040 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.071615 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:39253 (local address 127.0.0.1:55916)
0429 08:00:25.072315 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.072329 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.072415 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.073040 (+ 625us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.073043 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.073060 (+ 17us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:25.073504 (+ 444us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:25.073505 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:25.073512 (+ 7us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:25.073526 (+ 14us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:25.073880 (+ 354us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:25.073887 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:25.073894 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 08:00:25.073992 (+ 98us) client_negotiation.cc:241] Negotiation successful
0429 08:00:25.074025 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":57,"threads_started":1}
I20260429 08:00:25.075191 18041 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.071717 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.12.12.131:39633)
0429 08:00:25.072114 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.072120 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.072425 (+ 305us) server_negotiation.cc:408] Connection header received
0429 08:00:25.072527 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.072535 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.072678 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.072823 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.074252 (+ 1429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.074257 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.074264 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.074273 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.074305 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.074314 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.074427 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.074973 (+ 546us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.075026 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":131,"threads_started":1}
I20260429 08:00:25.077687 17963 tablet_service.cc:1511] Processing CreateTablet for tablet 62de87e9529546fb9f3420e2de3c90f5 (DEFAULT_TABLE table=test-table [id=06467a18f8e94ca287a1d6d39c3187ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:25.077689 17832 tablet_service.cc:1511] Processing CreateTablet for tablet 62de87e9529546fb9f3420e2de3c90f5 (DEFAULT_TABLE table=test-table [id=06467a18f8e94ca287a1d6d39c3187ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:25.077687 17698 tablet_service.cc:1511] Processing CreateTablet for tablet 62de87e9529546fb9f3420e2de3c90f5 (DEFAULT_TABLE table=test-table [id=06467a18f8e94ca287a1d6d39c3187ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:25.078579 17698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62de87e9529546fb9f3420e2de3c90f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:25.078599 17963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62de87e9529546fb9f3420e2de3c90f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:25.078600 17832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62de87e9529546fb9f3420e2de3c90f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:25.083655 18043 tablet_bootstrap.cc:492] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce: Bootstrap starting.
I20260429 08:00:25.084781 18044 tablet_bootstrap.cc:492] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Bootstrap starting.
I20260429 08:00:25.084916 18045 tablet_bootstrap.cc:492] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: Bootstrap starting.
I20260429 08:00:25.085783 18043 tablet_bootstrap.cc:654] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:25.086676 18043 log.cc:826] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:25.086860 18045 tablet_bootstrap.cc:654] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:25.086860 18044 tablet_bootstrap.cc:654] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:25.087677 18044 log.cc:826] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:25.087672 18045 log.cc:826] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:25.088497 18043 tablet_bootstrap.cc:492] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce: No bootstrap required, opened a new log
I20260429 08:00:25.088691 18043 ts_tablet_manager.cc:1403] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:00:25.089358 18044 tablet_bootstrap.cc:492] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: No bootstrap required, opened a new log
I20260429 08:00:25.089526 18045 tablet_bootstrap.cc:492] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: No bootstrap required, opened a new log
I20260429 08:00:25.089548 18044 ts_tablet_manager.cc:1403] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:00:25.089699 18045 ts_tablet_manager.cc:1403] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:00:25.091820 18044 raft_consensus.cc:359] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.091828 18043 raft_consensus.cc:359] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.092043 18043 raft_consensus.cc:385] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:25.092042 18044 raft_consensus.cc:385] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:25.092101 18043 raft_consensus.cc:740] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac400030a5d34b03a981bdef14f105ce, State: Initialized, Role: FOLLOWER
I20260429 08:00:25.092115 18044 raft_consensus.cc:740] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6d9a97873148318e2328a5a7d432c2, State: Initialized, Role: FOLLOWER
I20260429 08:00:25.092521 18043 consensus_queue.cc:260] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.092520 18044 consensus_queue.cc:260] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.092660 18045 raft_consensus.cc:359] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.092878 18045 raft_consensus.cc:385] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:25.092931 18045 raft_consensus.cc:740] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48404e8dbbfa4fd7820fa5374bdbbd59, State: Initialized, Role: FOLLOWER
I20260429 08:00:25.093314 18029 heartbeater.cc:499] Master 127.12.12.190:45809 was elected leader, sending a full tablet report...
I20260429 08:00:25.093322 18045 consensus_queue.cc:260] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.093398 18043 ts_tablet_manager.cc:1434] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 08:00:25.093665 17767 heartbeater.cc:499] Master 127.12.12.190:45809 was elected leader, sending a full tablet report...
I20260429 08:00:25.094010 18045 ts_tablet_manager.cc:1434] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260429 08:00:25.094288 17898 heartbeater.cc:499] Master 127.12.12.190:45809 was elected leader, sending a full tablet report...
I20260429 08:00:25.094369 18044 ts_tablet_manager.cc:1434] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:00:25.108244 18051 raft_consensus.cc:493] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:25.108414 18051 raft_consensus.cc:515] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.109915 18051 leader_election.cc:290] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f6d9a97873148318e2328a5a7d432c2 (127.12.12.131:39633), ac400030a5d34b03a981bdef14f105ce (127.12.12.129:39253)
I20260429 08:00:25.111703 17900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.110165 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:39633 (local address 127.12.12.130:34325)
0429 08:00:25.110294 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.110310 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.110383 (+ 73us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.110651 (+ 268us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.110654 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.110670 (+ 16us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:25.111184 (+ 514us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:25.111186 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:25.111198 (+ 12us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:25.111220 (+ 22us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:25.111481 (+ 261us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:25.111485 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:25.111488 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 08:00:25.111559 (+ 71us) client_negotiation.cc:241] Negotiation successful
0429 08:00:25.111586 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260429 08:00:25.111703 18041 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.110231 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:34325 (local address 127.12.12.131:39633)
0429 08:00:25.110363 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.110366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.110398 (+ 32us) server_negotiation.cc:408] Connection header received
0429 08:00:25.110479 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.110482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.110525 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.110616 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.111334 (+ 718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.111338 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.111340 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.111351 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.111371 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.111380 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.111484 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.111555 (+ 71us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.111586 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 08:00:25.112233 18042 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.110568 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:48623 (local address 127.12.12.129:39253)
0429 08:00:25.110727 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.110731 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.110910 (+ 179us) server_negotiation.cc:408] Connection header received
0429 08:00:25.111011 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.111014 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.111054 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.111135 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.111812 (+ 677us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.111815 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.111818 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.111828 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.111849 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.111857 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.111935 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.112084 (+ 149us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.112116 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 08:00:25.112243 18052 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.110451 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:39253 (local address 127.12.12.130:48623)
0429 08:00:25.110827 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.110838 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.110929 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.111175 (+ 246us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.111179 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.111195 (+ 16us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:25.111649 (+ 454us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:00:25.111652 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:25.111663 (+ 11us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:00:25.111692 (+ 29us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:25.111990 (+ 298us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:25.111994 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:25.111996 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 08:00:25.112074 (+ 78us) client_negotiation.cc:241] Negotiation successful
0429 08:00:25.112102 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":153,"threads_started":1}
I20260429 08:00:25.112488 17983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62de87e9529546fb9f3420e2de3c90f5" candidate_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6d9a97873148318e2328a5a7d432c2" is_pre_election: true
I20260429 08:00:25.112751 17983 raft_consensus.cc:2468] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48404e8dbbfa4fd7820fa5374bdbbd59 in term 0.
I20260429 08:00:25.113267 17785 leader_election.cc:304] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48404e8dbbfa4fd7820fa5374bdbbd59, 5f6d9a97873148318e2328a5a7d432c2; no voters:
I20260429 08:00:25.113505 18051 raft_consensus.cc:2804] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:25.113600 18051 raft_consensus.cc:493] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:25.113646 18051 raft_consensus.cc:3060] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:25.113554 17721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62de87e9529546fb9f3420e2de3c90f5" candidate_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac400030a5d34b03a981bdef14f105ce" is_pre_election: true
I20260429 08:00:25.114331 17721 raft_consensus.cc:2468] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48404e8dbbfa4fd7820fa5374bdbbd59 in term 0.
I20260429 08:00:25.115108 18051 raft_consensus.cc:515] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.115403 18051 leader_election.cc:290] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [CANDIDATE]: Term 1 election: Requested vote from peers 5f6d9a97873148318e2328a5a7d432c2 (127.12.12.131:39633), ac400030a5d34b03a981bdef14f105ce (127.12.12.129:39253)
I20260429 08:00:25.115785 17983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62de87e9529546fb9f3420e2de3c90f5" candidate_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6d9a97873148318e2328a5a7d432c2"
I20260429 08:00:25.115888 17721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62de87e9529546fb9f3420e2de3c90f5" candidate_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac400030a5d34b03a981bdef14f105ce"
I20260429 08:00:25.115967 17983 raft_consensus.cc:3060] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:25.116021 17721 raft_consensus.cc:3060] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:25.117086 17721 raft_consensus.cc:2468] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48404e8dbbfa4fd7820fa5374bdbbd59 in term 1.
I20260429 08:00:25.117403 17785 leader_election.cc:304] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48404e8dbbfa4fd7820fa5374bdbbd59, ac400030a5d34b03a981bdef14f105ce; no voters:
I20260429 08:00:25.117594 18051 raft_consensus.cc:2804] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:25.117615 17983 raft_consensus.cc:2468] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48404e8dbbfa4fd7820fa5374bdbbd59 in term 1.
I20260429 08:00:25.117818 18051 raft_consensus.cc:697] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 1 LEADER]: Becoming Leader. State: Replica: 48404e8dbbfa4fd7820fa5374bdbbd59, State: Running, Role: LEADER
I20260429 08:00:25.118135 18051 consensus_queue.cc:237] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } }
I20260429 08:00:25.121244 17577 catalog_manager.cc:5671] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48404e8dbbfa4fd7820fa5374bdbbd59 (127.12.12.130). New cstate: current_term: 1 leader_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48404e8dbbfa4fd7820fa5374bdbbd59" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:25.136891 18038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.135248 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.12.12.130:46321)
0429 08:00:25.135384 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.135388 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.135415 (+ 27us) server_negotiation.cc:408] Connection header received
0429 08:00:25.135509 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.135512 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.135562 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.135647 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.136360 (+ 713us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.136364 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.136367 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.136377 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.136407 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.136420 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.136511 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.136694 (+ 183us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.136732 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260429 08:00:25.139356 17899 tablet.cc:2404] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:25.142158 17983 raft_consensus.cc:1275] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 1 FOLLOWER]: Refusing update from remote peer 48404e8dbbfa4fd7820fa5374bdbbd59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:25.142164 17721 raft_consensus.cc:1275] T 62de87e9529546fb9f3420e2de3c90f5 P ac400030a5d34b03a981bdef14f105ce [term 1 FOLLOWER]: Refusing update from remote peer 48404e8dbbfa4fd7820fa5374bdbbd59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:25.142789 18053 consensus_queue.cc:1048] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac400030a5d34b03a981bdef14f105ce" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 39253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:25.142956 18051 consensus_queue.cc:1048] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6d9a97873148318e2328a5a7d432c2" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:25.150605 18058 mvcc.cc:204] Tried to move back new op lower bound from 7280433664575934464 to 7280433664487108608. Current Snapshot: MvccSnapshot[applied={T|T < 7280433664575934464}]
I20260429 08:00:25.158537 17577 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:25.158757 17577 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:25.161161 17577 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 67693de12ffb4be285be42fa16d10a21: Sending DeleteTablet for 3 replicas of tablet 62de87e9529546fb9f3420e2de3c90f5
I20260429 08:00:25.161827 17832 tablet_service.cc:1558] Processing DeleteTablet for tablet 62de87e9529546fb9f3420e2de3c90f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:25 UTC) from {username='slave'} at 127.0.0.1:50976
I20260429 08:00:25.162117 17963 tablet_service.cc:1558] Processing DeleteTablet for tablet 62de87e9529546fb9f3420e2de3c90f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:25 UTC) from {username='slave'} at 127.0.0.1:34012
I20260429 08:00:25.162464 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17639
I20260429 08:00:25.163327 18067 tablet_replica.cc:333] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: stopping tablet replica
I20260429 08:00:25.163580 18067 raft_consensus.cc:2243] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:25.163863 18067 raft_consensus.cc:2272] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:25.165344 18067 ts_tablet_manager.cc:1916] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:25.167788 18067 ts_tablet_manager.cc:1929] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:25.167879 18067 log.cc:1199] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/62de87e9529546fb9f3420e2de3c90f5
I20260429 08:00:25.168184 18066 tablet_replica.cc:333] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59: stopping tablet replica
I20260429 08:00:25.168206 18067 ts_tablet_manager.cc:1950] T 62de87e9529546fb9f3420e2de3c90f5 P 5f6d9a97873148318e2328a5a7d432c2: Deleting consensus metadata
I20260429 08:00:25.168582 18066 raft_consensus.cc:2243] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:25.169047 18066 raft_consensus.cc:2272] T 62de87e9529546fb9f3420e2de3c90f5 P 48404e8dbbfa4fd7820fa5374bdbbd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:25.169409 17563 catalog_manager.cc:5002] TS 5f6d9a97873148318e2328a5a7d432c2 (127.12.12.131:39633): tablet 62de87e9529546fb9f3420e2de3c90f5 (table test-table [id=06467a18f8e94ca287a1d6d39c3187ab]) successfully deleted
W20260429 08:00:25.170037 17563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.129:39253 (error 108)
I20260429 08:00:25.170204 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17770
I20260429 08:00:25.171876 18040 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.171501 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:39253 (local address 127.0.0.1:55930)
0429 08:00:25.171714 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.171773 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260429 08:00:25.172036 17563 catalog_manager.cc:4729] TS ac400030a5d34b03a981bdef14f105ce (127.12.12.129:39253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62de87e9529546fb9f3420e2de3c90f5: Network error: Client connection negotiation failed: client connection to 127.12.12.129:39253: connect: Connection refused (error 111)
I20260429 08:00:25.176311 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17901
I20260429 08:00:25.176956 18040 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.176509 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46321 (local address 127.0.0.1:50994)
0429 08:00:25.176690 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.176734 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.130:46321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260429 08:00:25.181926 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 17548
2026-04-29T08:00:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (831 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260429 08:00:25.203435 12338 mini_oidc.cc:150] Starting JWKS server
I20260429 08:00:25.204784 12338 webserver.cc:492] Webserver started at https://127.0.0.1:34907/ using document root <none> and password file <none>
I20260429 08:00:25.204981 12338 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 08:00:25.205227 12338 webserver.cc:492] Webserver started at http://127.0.0.1:41321/ using document root <none> and password file <none>
2026-04-29T08:00:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:25Z Disabled control of system clock
I20260429 08:00:25.213413 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:41737
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:41737
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34907/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/testchainca.cert with env {}
W20260429 08:00:25.314361 18074 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:25.314568 18074 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:25.314648 18074 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:25.318132 18074 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:25.318192 18074 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:25.318208 18074 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:25.318264 18074 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:25.318287 18074 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 08:00:25.318325 18074 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 08:00:25.318380 18074 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:34907/jwks.json
I20260429 08:00:25.322687 18074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:41737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:41737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34907/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:25.323732 18074 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:25.324584 18074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:25.330317 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:25.330348 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:25.330560 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:25.330626 18074 server_base.cc:1061] running on GCE node
I20260429 08:00:25.331091 18074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:25.331970 18074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:25.333176 18074 hybrid_clock.cc:648] HybridClock initialized: now 1777449625333154 us; error 39 us; skew 500 ppm
I20260429 08:00:25.334873 18074 webserver.cc:492] Webserver started at http://127.12.12.190:40091/ using document root <none> and password file <none>
I20260429 08:00:25.335448 18074 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:25.335531 18074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:25.335748 18074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:25.337486 18074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "7e6a52d44663452fb34eed4c5b1132d0"
format_stamp: "Formatted at 2026-04-29 08:00:25 on dist-test-slave-nhqz"
server_key: "a86056f6790c2e41c254d4e5eaa3755c"
server_key_iv: "2ac21375f1ac1e0d47e291f6cf1fce66"
server_key_version: "encryptionkey@0"
I20260429 08:00:25.338039 18074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "7e6a52d44663452fb34eed4c5b1132d0"
format_stamp: "Formatted at 2026-04-29 08:00:25 on dist-test-slave-nhqz"
server_key: "a86056f6790c2e41c254d4e5eaa3755c"
server_key_iv: "2ac21375f1ac1e0d47e291f6cf1fce66"
server_key_version: "encryptionkey@0"
I20260429 08:00:25.341218 18074 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:00:25.343241 18088 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:25.344177 18074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:25.344311 18074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "7e6a52d44663452fb34eed4c5b1132d0"
format_stamp: "Formatted at 2026-04-29 08:00:25 on dist-test-slave-nhqz"
server_key: "a86056f6790c2e41c254d4e5eaa3755c"
server_key_iv: "2ac21375f1ac1e0d47e291f6cf1fce66"
server_key_version: "encryptionkey@0"
I20260429 08:00:25.344420 18074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:25.355717 18074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:25.356333 18074 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:25.356529 18074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:25.363493 18074 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:41737
I20260429 08:00:25.363507 18140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:41737 every 8 connection(s)
I20260429 08:00:25.364450 18074 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:25.367086 18141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:25.368549 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18074
I20260429 08:00:25.368672 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:25.368955 12338 external_mini_cluster.cc:1468] Setting key 824a7cdc5326046be87efecfc0895f76
I20260429 08:00:25.372634 18141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0: Bootstrap starting.
I20260429 08:00:25.374711 18141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:25.375380 18141 log.cc:826] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:25.375977 18144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.370163 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.12.12.190:41737)
0429 08:00:25.370716 (+ 553us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.370724 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.370752 (+ 28us) server_negotiation.cc:408] Connection header received
0429 08:00:25.371205 (+ 453us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.371219 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.371537 (+ 318us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.371827 (+ 290us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:25.372346 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.373175 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:25.373825 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.374108 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:25.374601 (+ 493us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:25.374618 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:25.374628 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:25.374996 (+ 368us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:25.375026 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:25.375038 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:25.375133 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.375283 (+ 150us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.375414 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260429 08:00:25.377205 18141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0: No bootstrap required, opened a new log
I20260429 08:00:25.378485 12338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 08:00:25.379858 18141 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6a52d44663452fb34eed4c5b1132d0" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41737 } }
I20260429 08:00:25.380036 18141 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:25.380110 18141 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6a52d44663452fb34eed4c5b1132d0, State: Initialized, Role: FOLLOWER
I20260429 08:00:25.380518 18141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6a52d44663452fb34eed4c5b1132d0" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41737 } }
I20260429 08:00:25.380649 18141 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:25.380728 18141 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:25.380852 18141 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:25.381605 18141 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6a52d44663452fb34eed4c5b1132d0" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41737 } }
I20260429 08:00:25.381917 18141 leader_election.cc:304] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6a52d44663452fb34eed4c5b1132d0; no voters:
I20260429 08:00:25.382211 18141 leader_election.cc:290] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:25.382352 18146 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:25.382611 18146 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [term 1 LEADER]: Becoming Leader. State: Replica: 7e6a52d44663452fb34eed4c5b1132d0, State: Running, Role: LEADER
I20260429 08:00:25.382877 18146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6a52d44663452fb34eed4c5b1132d0" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41737 } }
I20260429 08:00:25.383491 18141 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:25.383913 18148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6a52d44663452fb34eed4c5b1132d0. Latest consensus state: current_term: 1 leader_uuid: "7e6a52d44663452fb34eed4c5b1132d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6a52d44663452fb34eed4c5b1132d0" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41737 } } }
I20260429 08:00:25.384076 18148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:25.384310 18147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6a52d44663452fb34eed4c5b1132d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6a52d44663452fb34eed4c5b1132d0" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41737 } } }
I20260429 08:00:25.384413 18147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:25.384831 18150 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:25.387651 18150 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:25.393079 18150 catalog_manager.cc:1357] Generated new cluster ID: 074fd62fda494b4cb91277661b96ab0b
I20260429 08:00:25.393169 18150 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:25.402611 18150 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:25.403685 18150 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:25.415009 18150 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 7e6a52d44663452fb34eed4c5b1132d0: Generated new TSK 0
I20260429 08:00:25.415772 18150 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:00:25.458691 18144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.428619 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.12.12.190:41737)
0429 08:00:25.428775 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.428779 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.428970 (+ 191us) server_negotiation.cc:408] Connection header received
0429 08:00:25.429203 (+ 233us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.429206 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.429258 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.429328 (+ 70us) server_negotiation.cc:227] Negotiated authn=JWT
0429 08:00:25.429822 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.430695 (+ 873us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:25.431582 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.431729 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:25.431793 (+ 64us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 08:00:25.458192 (+ 26399us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0429 08:00:25.458287 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:25.458476 (+ 189us) server_negotiation.cc:300] Negotiation successful
0429 08:00:25.458521 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60,"thread_start_us":135,"threads_started":1}
I20260429 08:00:25.470438 18144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:25.466521 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.12.12.190:41737)
0429 08:00:25.466652 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:25.466656 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:25.466740 (+ 84us) server_negotiation.cc:408] Connection header received
0429 08:00:25.466845 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:25.466849 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:25.466901 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:25.466991 (+ 90us) server_negotiation.cc:227] Negotiated authn=JWT
0429 08:00:25.467285 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.468068 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:25.468665 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.468904 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:25.468956 (+ 52us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 08:00:25.470096 (+ 1140us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0429 08:00:25.470304 (+ 208us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51798: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":55}
W20260429 08:00:25.470511 18144 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51798: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260429 08:00:25.471098 18177 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:25.466399 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41737 (local address 127.0.0.1:51798)
0429 08:00:25.466658 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:25.466671 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:25.466757 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:25.466993 (+ 236us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:25.466995 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:25.467000 (+ 5us) client_negotiation.cc:190] Negotiated authn=JWT
0429 08:00:25.467189 (+ 189us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:25.467195 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.468174 (+ 979us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:25.468176 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:25.468562 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:25.468568 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:25.468665 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:25.468682 (+ 17us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 08:00:25.470266 (+ 1584us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0429 08:00:25.470382 (+ 116us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.190:41737: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":115,"threads_started":1}
I20260429 08:00:40.479050 18183 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:40.475564 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.12.12.190:41737)
0429 08:00:40.475872 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:40.475878 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:40.475901 (+ 23us) server_negotiation.cc:408] Connection header received
0429 08:00:40.476087 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:40.476091 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:40.476146 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:40.476241 (+ 95us) server_negotiation.cc:227] Negotiated authn=JWT
0429 08:00:40.476622 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.477189 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:40.477857 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.478044 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:40.478098 (+ 54us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 08:00:40.478729 (+ 631us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0429 08:00:40.478900 (+ 171us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44024: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
W20260429 08:00:40.479111 18182 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:40.475416 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41737 (local address 127.0.0.1:44024)
0429 08:00:40.475818 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:40.475836 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:40.475967 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:40.476267 (+ 300us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:40.476270 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:40.476278 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0429 08:00:40.476495 (+ 217us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:40.476503 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.477320 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:40.477325 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:40.477734 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:40.477742 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.477872 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:40.477895 (+ 23us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 08:00:40.478859 (+ 964us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0429 08:00:40.478980 (+ 121us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.190:41737: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":116,"threads_started":1}
W20260429 08:00:40.479135 18183 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44024: JWT verification failed: token expired
W20260429 08:00:40.481662 18183 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:40.480857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.12.12.190:41737)
0429 08:00:40.480991 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:40.480995 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:40.481099 (+ 104us) server_negotiation.cc:408] Connection header received
0429 08:00:40.481242 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:40.481246 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:40.481291 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:40.481379 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:40.481566 (+ 187us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44034: BlockingRecv error: recv got EOF from 127.0.0.1:44034 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260429 08:00:40.485188 18193 client_negotiation.cc:376] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.12.12.190:41737
W20260429 08:00:40.485708 18183 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:40.484911 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.12.12.190:41737)
0429 08:00:40.485037 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:40.485041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:40.485202 (+ 161us) server_negotiation.cc:408] Connection header received
0429 08:00:40.485348 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:40.485351 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:40.485421 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:40.485499 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:40.485632 (+ 133us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44050: BlockingRecv error: recv got EOF from 127.0.0.1:44050 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 08:00:40.485998 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18074
2026-04-29T08:00:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260429 08:00:40.526466 12338 mini_oidc.cc:150] Starting JWKS server
I20260429 08:00:40.527468 12338 webserver.cc:492] Webserver started at https://127.0.0.1:41761/ using document root <none> and password file <none>
I20260429 08:00:40.527603 12338 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 08:00:40.527841 12338 webserver.cc:492] Webserver started at http://127.0.0.1:45205/ using document root <none> and password file <none>
2026-04-29T08:00:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:40Z Disabled control of system clock
I20260429 08:00:40.535656 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:37455
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:37455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41761/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/testchainca.cert with env {}
W20260429 08:00:40.636477 18199 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:40.636715 18199 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:40.636765 18199 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:40.640051 18199 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:40.640117 18199 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:40.640134 18199 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:40.640153 18199 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:40.640170 18199 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 08:00:40.640189 18199 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 08:00:40.640244 18199 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:41761/jwks.json
I20260429 08:00:40.644630 18199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:37455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:37455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41761/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:40.645723 18199 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:40.646569 18199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 08:00:40.652257 18199 server_base.cc:1061] running on GCE node
W20260429 08:00:40.652191 18207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:40.652181 18205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:40.652181 18204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:40.653083 18199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:40.653947 18199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:40.655105 18199 hybrid_clock.cc:648] HybridClock initialized: now 1777449640655081 us; error 37 us; skew 500 ppm
I20260429 08:00:40.657177 18199 webserver.cc:492] Webserver started at http://127.12.12.190:45533/ using document root <none> and password file <none>
I20260429 08:00:40.657780 18199 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:40.657840 18199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:40.658066 18199 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:40.659736 18199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "89a5c0050745460f8ad64f0ab5d3bf9f"
format_stamp: "Formatted at 2026-04-29 08:00:40 on dist-test-slave-nhqz"
server_key: "3ec956312b6252933d6d8c9bbfee3330"
server_key_iv: "4ac93e28c9d698bccd63b44efb0fe868"
server_key_version: "encryptionkey@0"
I20260429 08:00:40.660218 18199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "89a5c0050745460f8ad64f0ab5d3bf9f"
format_stamp: "Formatted at 2026-04-29 08:00:40 on dist-test-slave-nhqz"
server_key: "3ec956312b6252933d6d8c9bbfee3330"
server_key_iv: "4ac93e28c9d698bccd63b44efb0fe868"
server_key_version: "encryptionkey@0"
I20260429 08:00:40.663539 18199 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 08:00:40.665803 18213 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:40.666802 18199 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:40.666935 18199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "89a5c0050745460f8ad64f0ab5d3bf9f"
format_stamp: "Formatted at 2026-04-29 08:00:40 on dist-test-slave-nhqz"
server_key: "3ec956312b6252933d6d8c9bbfee3330"
server_key_iv: "4ac93e28c9d698bccd63b44efb0fe868"
server_key_version: "encryptionkey@0"
I20260429 08:00:40.667038 18199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:40.678411 18199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:40.679042 18199 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:40.679227 18199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:40.686652 18199 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:37455
I20260429 08:00:40.686650 18265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:37455 every 8 connection(s)
I20260429 08:00:40.687644 18199 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:40.690336 18266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:40.690387 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18199
I20260429 08:00:40.690505 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:40.690779 12338 external_mini_cluster.cc:1468] Setting key 14e37c1b014878b91747a6b195c4191a
I20260429 08:00:40.696175 18266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f: Bootstrap starting.
I20260429 08:00:40.698365 18266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:40.699052 18266 log.cc:826] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:40.700976 18266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f: No bootstrap required, opened a new log
I20260429 08:00:40.702337 18269 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:40.691993 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.12.12.190:37455)
0429 08:00:40.692557 (+ 564us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:40.692564 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:40.692596 (+ 32us) server_negotiation.cc:408] Connection header received
0429 08:00:40.693613 (+ 1017us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:40.693633 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:40.693956 (+ 323us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:40.694277 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:40.694776 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.695500 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:40.696164 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.696455 (+ 291us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:40.700794 (+ 4339us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:40.700858 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:40.700877 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:40.701294 (+ 417us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:40.701331 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:40.701346 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:40.701456 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:40.701646 (+ 190us) server_negotiation.cc:300] Negotiation successful
0429 08:00:40.701801 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":114,"threads_started":1}
I20260429 08:00:40.703616 18266 raft_consensus.cc:359] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37455 } }
I20260429 08:00:40.703835 18266 raft_consensus.cc:385] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:40.703912 18266 raft_consensus.cc:740] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a5c0050745460f8ad64f0ab5d3bf9f, State: Initialized, Role: FOLLOWER
I20260429 08:00:40.704337 18266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37455 } }
I20260429 08:00:40.704497 18266 raft_consensus.cc:399] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:40.704586 18266 raft_consensus.cc:493] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:40.704680 18266 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:40.705063 12338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 08:00:40.705653 18266 raft_consensus.cc:515] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37455 } }
I20260429 08:00:40.706025 18266 leader_election.cc:304] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89a5c0050745460f8ad64f0ab5d3bf9f; no voters:
I20260429 08:00:40.706377 18266 leader_election.cc:290] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:40.706429 18271 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:40.706602 18271 raft_consensus.cc:697] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [term 1 LEADER]: Becoming Leader. State: Replica: 89a5c0050745460f8ad64f0ab5d3bf9f, State: Running, Role: LEADER
I20260429 08:00:40.706934 18271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37455 } }
I20260429 08:00:40.707502 18266 sys_catalog.cc:565] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:40.708077 18271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37455 } } }
I20260429 08:00:40.708278 18271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:40.708468 18272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89a5c0050745460f8ad64f0ab5d3bf9f. Latest consensus state: current_term: 1 leader_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a5c0050745460f8ad64f0ab5d3bf9f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 37455 } } }
I20260429 08:00:40.708595 18272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:40.708709 18277 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:40.711659 18277 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:40.716553 18277 catalog_manager.cc:1357] Generated new cluster ID: 80467e90f35546089bd9d011c2db12a8
I20260429 08:00:40.716624 18277 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:40.735355 18277 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:40.736255 18277 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:40.749826 18277 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 89a5c0050745460f8ad64f0ab5d3bf9f: Generated new TSK 0
I20260429 08:00:40.750447 18277 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260429 08:00:40.773782 18296 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260429 08:00:40.773877 18296 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260429 08:00:40.774410 18269 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:40.758508 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.12.12.190:37455)
0429 08:00:40.758640 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:40.758645 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:40.758782 (+ 137us) server_negotiation.cc:408] Connection header received
0429 08:00:40.758923 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:40.758926 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:40.758986 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:40.759091 (+ 105us) server_negotiation.cc:227] Negotiated authn=JWT
0429 08:00:40.759438 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.760250 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:40.760923 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.761118 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:40.761208 (+ 90us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 08:00:40.774001 (+ 12793us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 08:00:40.774269 (+ 268us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260429 08:00:40.774492 18269 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260429 08:00:40.774519 18295 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:40.758411 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:37455 (local address 127.0.0.1:58510)
0429 08:00:40.758679 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:40.758692 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:40.758799 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:40.759100 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:40.759102 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:40.759108 (+ 6us) client_negotiation.cc:190] Negotiated authn=JWT
0429 08:00:40.759329 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:40.759336 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.760358 (+ 1022us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:40.760361 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:40.760765 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:40.760773 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.760933 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:40.760952 (+ 19us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 08:00:40.774198 (+ 13246us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 08:00:40.774355 (+ 157us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.190:37455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":112,"threads_started":1}
I20260429 08:00:40.775153 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18199
2026-04-29T08:00:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260429 08:00:40.815135 12338 mini_oidc.cc:150] Starting JWKS server
I20260429 08:00:40.816046 12338 webserver.cc:492] Webserver started at https://127.0.0.1:44813/ using document root <none> and password file <none>
I20260429 08:00:40.816148 12338 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 08:00:40.816393 12338 webserver.cc:492] Webserver started at http://127.0.0.1:39699/ using document root <none> and password file <none>
2026-04-29T08:00:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:40Z Disabled control of system clock
I20260429 08:00:40.824322 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:46437
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:35811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:46437
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44813/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260429 08:00:40.923167 18302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:40.923461 18302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:40.923574 18302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:40.926954 18302 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:40.927064 18302 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:40.927096 18302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:40.927139 18302 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:40.927174 18302 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 08:00:40.927235 18302 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 08:00:40.927263 18302 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:44813/jwks.json
I20260429 08:00:40.931330 18302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:35811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:46437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44813/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:40.932535 18302 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:40.933712 18302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:40.939188 18308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:40.939175 18310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:40.939177 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:40.939561 18302 server_base.cc:1061] running on GCE node
I20260429 08:00:40.940053 18302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:40.940963 18302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:40.942107 18302 hybrid_clock.cc:648] HybridClock initialized: now 1777449640942092 us; error 31 us; skew 500 ppm
I20260429 08:00:40.943974 18302 webserver.cc:492] Webserver started at http://127.12.12.190:36043/ using document root <none> and password file <none>
I20260429 08:00:40.944523 18302 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:40.944590 18302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:40.944892 18302 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:40.946511 18302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "293e84f9deb74fe6a61b831e40d4ed56"
format_stamp: "Formatted at 2026-04-29 08:00:40 on dist-test-slave-nhqz"
server_key: "3b59175f170d84856e370be6074a80e4"
server_key_iv: "9b2edca59b35798d0619e5ec8210de8b"
server_key_version: "encryptionkey@0"
I20260429 08:00:40.947019 18302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "293e84f9deb74fe6a61b831e40d4ed56"
format_stamp: "Formatted at 2026-04-29 08:00:40 on dist-test-slave-nhqz"
server_key: "3b59175f170d84856e370be6074a80e4"
server_key_iv: "9b2edca59b35798d0619e5ec8210de8b"
server_key_version: "encryptionkey@0"
I20260429 08:00:40.950340 18302 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 08:00:40.952533 18316 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:40.953542 18302 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:40.953673 18302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "293e84f9deb74fe6a61b831e40d4ed56"
format_stamp: "Formatted at 2026-04-29 08:00:40 on dist-test-slave-nhqz"
server_key: "3b59175f170d84856e370be6074a80e4"
server_key_iv: "9b2edca59b35798d0619e5ec8210de8b"
server_key_version: "encryptionkey@0"
I20260429 08:00:40.953773 18302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:40.972659 18302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:40.973379 18302 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:40.973592 18302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:40.980749 18302 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:46437
I20260429 08:00:40.980764 18368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:46437 every 8 connection(s)
I20260429 08:00:40.981761 18302 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:40.984424 18369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:40.989400 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18302
I20260429 08:00:40.989547 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:40.989779 12338 external_mini_cluster.cc:1468] Setting key 11733d753d27aeaf441d21cc2d60aace
I20260429 08:00:40.989841 18369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56: Bootstrap starting.
I20260429 08:00:40.992255 18369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:40.993155 18369 log.cc:826] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:40.995185 18369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56: No bootstrap required, opened a new log
I20260429 08:00:40.997115 18372 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:40.990927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.12.12.190:46437)
0429 08:00:40.991449 (+ 522us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:40.991456 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:40.991489 (+ 33us) server_negotiation.cc:408] Connection header received
0429 08:00:40.991994 (+ 505us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:40.992009 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:40.992345 (+ 336us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:40.992659 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:40.993297 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.994043 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:40.994895 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:40.995113 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:40.995674 (+ 561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:40.995692 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:40.995702 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:40.996044 (+ 342us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:40.996076 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:40.996088 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:40.996165 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:40.996347 (+ 182us) server_negotiation.cc:300] Negotiation successful
0429 08:00:40.996523 (+ 176us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
I20260429 08:00:40.997893 18369 raft_consensus.cc:359] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293e84f9deb74fe6a61b831e40d4ed56" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46437 } }
I20260429 08:00:40.998122 18369 raft_consensus.cc:385] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:40.998201 18369 raft_consensus.cc:740] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293e84f9deb74fe6a61b831e40d4ed56, State: Initialized, Role: FOLLOWER
I20260429 08:00:40.998597 18369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293e84f9deb74fe6a61b831e40d4ed56" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46437 } }
I20260429 08:00:40.998733 18369 raft_consensus.cc:399] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:40.998812 18369 raft_consensus.cc:493] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:40.998898 18369 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:40.999874 18369 raft_consensus.cc:515] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293e84f9deb74fe6a61b831e40d4ed56" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46437 } }
I20260429 08:00:40.999943 12338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 08:00:41.000264 18369 leader_election.cc:304] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 293e84f9deb74fe6a61b831e40d4ed56; no voters:
I20260429 08:00:41.000546 18369 leader_election.cc:290] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:41.000669 18374 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:41.000937 18374 raft_consensus.cc:697] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [term 1 LEADER]: Becoming Leader. State: Replica: 293e84f9deb74fe6a61b831e40d4ed56, State: Running, Role: LEADER
I20260429 08:00:41.001236 18374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293e84f9deb74fe6a61b831e40d4ed56" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46437 } }
I20260429 08:00:41.001756 18369 sys_catalog.cc:565] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:41.002812 18376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "293e84f9deb74fe6a61b831e40d4ed56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293e84f9deb74fe6a61b831e40d4ed56" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46437 } } }
I20260429 08:00:41.002930 18376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:41.003607 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 293e84f9deb74fe6a61b831e40d4ed56. Latest consensus state: current_term: 1 leader_uuid: "293e84f9deb74fe6a61b831e40d4ed56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "293e84f9deb74fe6a61b831e40d4ed56" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 46437 } } }
I20260429 08:00:41.003760 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:41.003952 18383 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:41.006436 18383 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:41.011406 18383 catalog_manager.cc:1357] Generated new cluster ID: 6ff6735594fa414a9888304470d76f4e
I20260429 08:00:41.011503 18383 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:41.023308 18383 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:41.024128 18383 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:41.029392 18383 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 293e84f9deb74fe6a61b831e40d4ed56: Generated new TSK 0
I20260429 08:00:41.029924 18383 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260429 08:00:41.057504 18399 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260429 08:00:41.057579 18399 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260429 08:00:41.059023 18372 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:41.037150 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.12.12.190:46437)
0429 08:00:41.037282 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:41.037285 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:41.037425 (+ 140us) server_negotiation.cc:408] Connection header received
0429 08:00:41.037602 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:41.037605 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:41.037655 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:41.037728 (+ 73us) server_negotiation.cc:227] Negotiated authn=JWT
0429 08:00:41.038200 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.038718 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:41.039683 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.039821 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:41.039895 (+ 74us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 08:00:41.058622 (+ 18727us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 08:00:41.058896 (+ 274us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260429 08:00:41.059106 18372 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260429 08:00:41.059456 18398 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:41.037006 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:46437 (local address 127.0.0.1:40456)
0429 08:00:41.037334 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:41.037350 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:41.037470 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:41.037764 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:41.037768 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:41.037776 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0429 08:00:41.038087 (+ 311us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:41.038096 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.038847 (+ 751us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:41.038851 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:41.039565 (+ 714us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:41.039576 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.039689 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:41.039714 (+ 25us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 08:00:41.058830 (+ 19116us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 08:00:41.059019 (+ 189us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.190:46437: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":134,"threads_started":1}
I20260429 08:00:41.060111 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18302
2026-04-29T08:00:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-04-29T08:00:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:41Z Disabled control of system clock
I20260429 08:00:41.107060 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:44393
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:44393
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/testchainca.cert with env {}
W20260429 08:00:41.206430 18403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:41.206625 18403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:41.206727 18403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:41.209897 18403 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:41.209955 18403 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:41.209971 18403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:41.209990 18403 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:41.210006 18403 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 08:00:41.210023 18403 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 08:00:41.210036 18403 flags.cc:432] Enabled experimental flag: --jwks_url=default.url
I20260429 08:00:41.214330 18403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:44393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:44393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:41.215366 18403 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:41.216142 18403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:41.221719 18411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:41.221833 18408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:41.221740 18409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:41.222034 18403 server_base.cc:1061] running on GCE node
I20260429 08:00:41.222442 18403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:41.223253 18403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:41.224403 18403 hybrid_clock.cc:648] HybridClock initialized: now 1777449641224377 us; error 40 us; skew 500 ppm
I20260429 08:00:41.226248 18403 webserver.cc:492] Webserver started at http://127.12.12.190:44467/ using document root <none> and password file <none>
I20260429 08:00:41.226794 18403 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:41.226848 18403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:41.227051 18403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:41.228685 18403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "833584840ca64899b5bb5a744d37f31b"
format_stamp: "Formatted at 2026-04-29 08:00:41 on dist-test-slave-nhqz"
server_key: "c25052f0640298f23251bd21ad260315"
server_key_iv: "9eae469684f8e5aa9c59c73f37f968f4"
server_key_version: "encryptionkey@0"
I20260429 08:00:41.229212 18403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "833584840ca64899b5bb5a744d37f31b"
format_stamp: "Formatted at 2026-04-29 08:00:41 on dist-test-slave-nhqz"
server_key: "c25052f0640298f23251bd21ad260315"
server_key_iv: "9eae469684f8e5aa9c59c73f37f968f4"
server_key_version: "encryptionkey@0"
I20260429 08:00:41.232244 18403 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 08:00:41.234521 18417 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:41.235630 18403 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:41.235718 18403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "833584840ca64899b5bb5a744d37f31b"
format_stamp: "Formatted at 2026-04-29 08:00:41 on dist-test-slave-nhqz"
server_key: "c25052f0640298f23251bd21ad260315"
server_key_iv: "9eae469684f8e5aa9c59c73f37f968f4"
server_key_version: "encryptionkey@0"
I20260429 08:00:41.235788 18403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:41.253716 18403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:41.254267 18403 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:41.254405 18403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:41.260500 18403 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:44393
I20260429 08:00:41.260553 18469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:44393 every 8 connection(s)
I20260429 08:00:41.261428 18403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:41.262250 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18403
I20260429 08:00:41.262385 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:41.262653 12338 external_mini_cluster.cc:1468] Setting key e87a78da4e28b2d8187b970b870c293f
I20260429 08:00:41.264508 18470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:41.269187 18470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b: Bootstrap starting.
I20260429 08:00:41.270279 18473 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:41.263805 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.12.12.190:44393)
0429 08:00:41.264351 (+ 546us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:41.264358 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:41.264389 (+ 31us) server_negotiation.cc:408] Connection header received
0429 08:00:41.265092 (+ 703us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:41.265110 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:41.265420 (+ 310us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:41.265736 (+ 316us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:41.266252 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.267377 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:41.268035 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.268312 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:41.268846 (+ 534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:41.268892 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:41.268903 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:00:41.269254 (+ 351us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:41.269286 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:00:41.269299 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:41.269417 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:41.269598 (+ 181us) server_negotiation.cc:300] Negotiation successful
0429 08:00:41.269750 (+ 152us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260429 08:00:41.271389 18470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:41.272151 18470 log.cc:826] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:41.272765 12338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 08:00:41.274006 18470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b: No bootstrap required, opened a new log
I20260429 08:00:41.276322 18470 raft_consensus.cc:359] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833584840ca64899b5bb5a744d37f31b" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44393 } }
I20260429 08:00:41.276533 18470 raft_consensus.cc:385] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:41.276628 18470 raft_consensus.cc:740] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833584840ca64899b5bb5a744d37f31b, State: Initialized, Role: FOLLOWER
I20260429 08:00:41.277092 18470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833584840ca64899b5bb5a744d37f31b" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44393 } }
I20260429 08:00:41.277240 18470 raft_consensus.cc:399] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:41.277324 18470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:41.277433 18470 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:41.278267 18470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833584840ca64899b5bb5a744d37f31b" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44393 } }
I20260429 08:00:41.278589 18470 leader_election.cc:304] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833584840ca64899b5bb5a744d37f31b; no voters:
I20260429 08:00:41.278885 18470 leader_election.cc:290] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:41.278961 18475 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:41.279256 18475 raft_consensus.cc:697] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [term 1 LEADER]: Becoming Leader. State: Replica: 833584840ca64899b5bb5a744d37f31b, State: Running, Role: LEADER
I20260429 08:00:41.279525 18475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833584840ca64899b5bb5a744d37f31b" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44393 } }
I20260429 08:00:41.280050 18470 sys_catalog.cc:565] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:41.281260 18477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833584840ca64899b5bb5a744d37f31b. Latest consensus state: current_term: 1 leader_uuid: "833584840ca64899b5bb5a744d37f31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833584840ca64899b5bb5a744d37f31b" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44393 } } }
I20260429 08:00:41.281366 18477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:41.281455 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "833584840ca64899b5bb5a744d37f31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833584840ca64899b5bb5a744d37f31b" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 44393 } } }
I20260429 08:00:41.281599 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:41.281704 18484 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:41.284058 18484 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:41.288625 18484 catalog_manager.cc:1357] Generated new cluster ID: d19ed022c2684809864d1a3945211a37
I20260429 08:00:41.288693 18484 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:41.308179 18484 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:41.309369 18484 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:41.324542 18484 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 833584840ca64899b5bb5a744d37f31b: Generated new TSK 0
I20260429 08:00:41.325331 18484 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:00:41.479174 18473 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:41.341161 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.12.12.190:44393)
0429 08:00:41.341300 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:41.341304 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:41.341446 (+ 142us) server_negotiation.cc:408] Connection header received
0429 08:00:41.341582 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:41.341586 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:41.341644 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:41.341756 (+ 112us) server_negotiation.cc:227] Negotiated authn=JWT
0429 08:00:41.342062 (+ 306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.342875 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:41.343528 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.343726 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:41.343811 (+ 85us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 08:00:41.478807 (+134996us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0429 08:00:41.479052 (+ 245us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56616: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260429 08:00:41.479288 18473 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56616: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260429 08:00:41.479422 18499 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:41.341044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:44393 (local address 127.0.0.1:56616)
0429 08:00:41.341365 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:41.341377 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:41.341463 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:41.341757 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:41.341759 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:41.341765 (+ 6us) client_negotiation.cc:190] Negotiated authn=JWT
0429 08:00:41.341963 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:41.341969 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.342976 (+ 1007us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:41.342978 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:41.343422 (+ 444us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:41.343430 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:41.343520 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:41.343539 (+ 19us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 08:00:41.479043 (+135504us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0429 08:00:41.479199 (+ 156us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.12.190:44393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260429 08:00:41.480144 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18403
2026-04-29T08:00:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (400 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (110 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (111 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:41 dist-test-slave-nhqz krb5kdc[18511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:41 dist-test-slave-nhqz krb5kdc[18511](info): set up 2 sockets
Apr 29 08:00:41 dist-test-slave-nhqz krb5kdc[18511](info): commencing operation
krb5kdc: starting...
W20260429 08:00:43.686167 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.964s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:43 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:43.825799 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:33377
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:43.925685 18527 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:43.925911 18527 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:43.926002 18527 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:43.929412 18527 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:43.929476 18527 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:43.929499 18527 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:43.929517 18527 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:43.929534 18527 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:43.933945 18527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:43.935052 18527 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:43.935920 18527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:43.941529 18532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:43.941536 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:43.941594 18527 server_base.cc:1061] running on GCE node
W20260429 08:00:43.941768 18533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:43.942267 18527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:43.943197 18527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:43.944407 18527 hybrid_clock.cc:648] HybridClock initialized: now 1777449643944394 us; error 40 us; skew 500 ppm
Apr 29 08:00:43 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449643, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:43.947103 18527 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:00:43.948164 18527 webserver.cc:492] Webserver started at http://127.12.12.190:44117/ using document root <none> and password file <none>
I20260429 08:00:43.948735 18527 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:43.948815 18527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:43.949041 18527 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:43.950706 18527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "5cfb10134b824fe083790f3789ce8262"
format_stamp: "Formatted at 2026-04-29 08:00:43 on dist-test-slave-nhqz"
server_key: "c2f14cebb9593a66015e25ac1cea7cab"
server_key_iv: "4714f77d094bf540748cc352398da9da"
server_key_version: "encryptionkey@0"
I20260429 08:00:43.951202 18527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "5cfb10134b824fe083790f3789ce8262"
format_stamp: "Formatted at 2026-04-29 08:00:43 on dist-test-slave-nhqz"
server_key: "c2f14cebb9593a66015e25ac1cea7cab"
server_key_iv: "4714f77d094bf540748cc352398da9da"
server_key_version: "encryptionkey@0"
I20260429 08:00:43.954501 18527 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:43.956548 18542 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:43.957567 18527 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:43.957700 18527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "5cfb10134b824fe083790f3789ce8262"
format_stamp: "Formatted at 2026-04-29 08:00:43 on dist-test-slave-nhqz"
server_key: "c2f14cebb9593a66015e25ac1cea7cab"
server_key_iv: "4714f77d094bf540748cc352398da9da"
server_key_version: "encryptionkey@0"
I20260429 08:00:43.957803 18527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:44.006316 18527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:44.009419 18527 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:44.009639 18527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:44.017030 18527 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:33377
I20260429 08:00:44.017030 18594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:33377 every 8 connection(s)
I20260429 08:00:44.018090 18527 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:44.020798 18595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:44.022112 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18527
I20260429 08:00:44.022236 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:44.022528 12338 external_mini_cluster.cc:1468] Setting key e8db66c19373104c2b740f8636c05681
I20260429 08:00:44.026732 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262: Bootstrap starting.
I20260429 08:00:44.028988 18595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:44.029814 18595 log.cc:826] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262: Log is configured to *not* fsync() on all Append() calls
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:44.031855 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262: No bootstrap required, opened a new log
I20260429 08:00:44.034780 18598 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.023873 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.12.12.190:33377)
0429 08:00:44.024277 (+ 404us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.024286 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.024318 (+ 32us) server_negotiation.cc:408] Connection header received
0429 08:00:44.024872 (+ 554us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.024889 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.025149 (+ 260us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.025448 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.026394 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.027191 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.027871 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.028154 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.030688 (+ 2534us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.030704 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.030714 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.030740 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.032419 (+ 1679us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.032898 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.032902 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.032907 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.032972 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.033205 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.033207 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.033208 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.033479 (+ 271us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.033622 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.033952 (+ 330us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.034186 (+ 234us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
I20260429 08:00:44.034761 18595 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfb10134b824fe083790f3789ce8262" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33377 } }
I20260429 08:00:44.035044 18595 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:44.035135 18595 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfb10134b824fe083790f3789ce8262, State: Initialized, Role: FOLLOWER
I20260429 08:00:44.035547 18595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfb10134b824fe083790f3789ce8262" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33377 } }
I20260429 08:00:44.035694 18595 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:44.035773 18595 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:44.035871 18595 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:44.037050 18595 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfb10134b824fe083790f3789ce8262" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33377 } }
I20260429 08:00:44.037382 18595 leader_election.cc:304] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cfb10134b824fe083790f3789ce8262; no voters:
I20260429 08:00:44.037662 18595 leader_election.cc:290] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:44.037828 18600 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:44.038057 18600 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [term 1 LEADER]: Becoming Leader. State: Replica: 5cfb10134b824fe083790f3789ce8262, State: Running, Role: LEADER
I20260429 08:00:44.038357 18600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfb10134b824fe083790f3789ce8262" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33377 } }
I20260429 08:00:44.038789 18595 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:44.039898 18601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cfb10134b824fe083790f3789ce8262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfb10134b824fe083790f3789ce8262" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33377 } } }
I20260429 08:00:44.040014 18601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:44.040282 18602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfb10134b824fe083790f3789ce8262. Latest consensus state: current_term: 1 leader_uuid: "5cfb10134b824fe083790f3789ce8262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfb10134b824fe083790f3789ce8262" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33377 } } }
I20260429 08:00:44.040387 18609 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:44.040445 18602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:44.043551 18609 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:44.049131 18609 catalog_manager.cc:1357] Generated new cluster ID: 010e9f6b95af4cb8bc52afd4cc340754
I20260429 08:00:44.049211 18609 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:44.067911 18609 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:44.069109 18609 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:44.079707 18609 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5cfb10134b824fe083790f3789ce8262: Generated new TSK 0
I20260429 08:00:44.080397 18609 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:44.136659 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:33377
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:44.241868 18623 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:44.242110 18623 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:44.242197 18623 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:44.245630 18623 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:44.245738 18623 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:44.245854 18623 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:44.250187 18623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:33377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:44.251303 18623 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:44.252202 18623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:44.258739 18628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:44.258832 18629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:44.258754 18631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:44.258952 18623 server_base.cc:1061] running on GCE node
I20260429 08:00:44.259294 18623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:44.259857 18623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:44.261026 18623 hybrid_clock.cc:648] HybridClock initialized: now 1777449644261014 us; error 33 us; skew 500 ppm
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.263720 18623 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:00:44.264685 18623 webserver.cc:492] Webserver started at http://127.12.12.129:42293/ using document root <none> and password file <none>
I20260429 08:00:44.265244 18623 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:44.265300 18623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:44.265453 18623 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:44.267035 18623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "afe087fa961e4e8a8b7703c3e9a2af03"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "c22ab60b1f67f227fc555b80a180099a"
server_key_iv: "0d744662dbb67840c1ef4d85f8ebabb3"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.267472 18623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "afe087fa961e4e8a8b7703c3e9a2af03"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "c22ab60b1f67f227fc555b80a180099a"
server_key_iv: "0d744662dbb67840c1ef4d85f8ebabb3"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.270648 18623 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:44.272641 18638 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.273624 18623 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:44.273758 18623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "afe087fa961e4e8a8b7703c3e9a2af03"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "c22ab60b1f67f227fc555b80a180099a"
server_key_iv: "0d744662dbb67840c1ef4d85f8ebabb3"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.273862 18623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:44.289284 18623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:44.292310 18623 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:44.292526 18623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:44.293197 18623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:44.294111 18623 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:44.294185 18623 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.294255 18623 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:44.294312 18623 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.303442 18623 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:42059
I20260429 08:00:44.303467 18751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:42059 every 8 connection(s)
I20260429 08:00:44.304423 18623 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:44.313006 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18623
I20260429 08:00:44.313134 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:44.313452 12338 external_mini_cluster.cc:1468] Setting key e8009c21354dd80dd67f71aa8baa23b0
I20260429 08:00:44.316694 18598 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.306272 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:59261 (local address 127.12.12.190:33377)
0429 08:00:44.306453 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.306458 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.307086 (+ 628us) server_negotiation.cc:408] Connection header received
0429 08:00:44.307892 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.307895 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.307943 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.308020 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.309573 (+ 1553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.310077 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.310686 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.310814 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.312887 (+ 2073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.312909 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.312915 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.312948 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.314626 (+ 1678us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.315213 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.315217 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.315218 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.315272 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.315572 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.315578 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.315583 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.315782 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.315886 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.316319 (+ 433us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.316458 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 08:00:44.317432 18754 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.306540 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33377 (local address 127.12.12.129:59261)
0429 08:00:44.306943 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:44.306975 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:44.307713 (+ 738us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:44.308169 (+ 456us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:44.308176 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:44.308561 (+ 385us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:44.309423 (+ 862us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:44.309436 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.310182 (+ 746us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.310185 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:44.310587 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:44.310592 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.310738 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.311263 (+ 525us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:44.311280 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:44.312689 (+ 1409us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:44.314800 (+ 2111us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:44.314806 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:44.314819 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:44.315112 (+ 293us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:44.315368 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:44.315371 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:44.315372 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:44.315470 (+ 98us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:44.315884 (+ 414us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:44.315887 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:44.316084 (+ 197us) client_negotiation.cc:770] Sending connection context
0429 08:00:44.316271 (+ 187us) client_negotiation.cc:241] Negotiation successful
0429 08:00:44.316456 (+ 185us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260429 08:00:44.318662 18752 heartbeater.cc:344] Connected to a master server at 127.12.12.190:33377
I20260429 08:00:44.318933 18752 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:44.319442 18752 heartbeater.cc:507] Master 127.12.12.190:33377 requested a full tablet report, sending...
I20260429 08:00:44.321172 18559 ts_manager.cc:194] Registered new tserver with Master: afe087fa961e4e8a8b7703c3e9a2af03 (127.12.12.129:42059)
I20260429 08:00:44.322875 18559 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:59261
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:44.363538 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:33377
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:44.468974 18759 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:44.469292 18759 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:44.469383 18759 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:44.472671 18759 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:44.472764 18759 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:44.472903 18759 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:44.477126 18759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:33377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:44.478299 18759 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:44.479123 18759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:44.485427 18764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:44.485527 18765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:44.485687 18759 server_base.cc:1061] running on GCE node
W20260429 08:00:44.485527 18767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:44.486083 18759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:44.486662 18759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:44.487841 18759 hybrid_clock.cc:648] HybridClock initialized: now 1777449644487828 us; error 26 us; skew 500 ppm
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.490471 18759 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:00:44.491482 18759 webserver.cc:492] Webserver started at http://127.12.12.130:34111/ using document root <none> and password file <none>
I20260429 08:00:44.491961 18759 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:44.492005 18759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:44.492153 18759 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:44.493834 18759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "66e6f21f2e5e4f0db32183663b921491"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "157b77e1631e83ad765aaf379a334111"
server_key_iv: "0807713e0a7275e180126da627cba371"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.494254 18759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "66e6f21f2e5e4f0db32183663b921491"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "157b77e1631e83ad765aaf379a334111"
server_key_iv: "0807713e0a7275e180126da627cba371"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.497520 18759 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 08:00:44.499526 18774 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.500460 18759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:00:44.500581 18759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "66e6f21f2e5e4f0db32183663b921491"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "157b77e1631e83ad765aaf379a334111"
server_key_iv: "0807713e0a7275e180126da627cba371"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.500690 18759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:44.512535 18759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:44.515033 18759 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:44.515231 18759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:44.515777 18759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:44.516633 18759 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:44.516678 18759 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.516716 18759 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:44.516754 18759 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.526158 18759 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:39451
I20260429 08:00:44.526171 18887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:39451 every 8 connection(s)
I20260429 08:00:44.527120 18759 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:44.529814 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18759
I20260429 08:00:44.529937 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:44.530210 12338 external_mini_cluster.cc:1468] Setting key 3f515dcb4934a9875c70851db0196b3b
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:00:44.539814 18598 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.528912 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:55585 (local address 127.12.12.190:33377)
0429 08:00:44.529056 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.529060 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.529727 (+ 667us) server_negotiation.cc:408] Connection header received
0429 08:00:44.530690 (+ 963us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.530694 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.530753 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.530860 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.532477 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.533042 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.533687 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.533865 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.536217 (+ 2352us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.536241 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.536243 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.536269 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.537766 (+ 1497us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.538272 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.538276 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.538278 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.538328 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.538617 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.538621 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.538623 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.538804 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.538908 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.539395 (+ 487us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.539517 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 08:00:44.540381 18890 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.529184 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33377 (local address 127.12.12.130:55585)
0429 08:00:44.529592 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:44.529623 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:44.530491 (+ 868us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:44.531017 (+ 526us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:44.531025 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:44.531395 (+ 370us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:44.532330 (+ 935us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:44.532342 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.533162 (+ 820us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.533171 (+ 9us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:44.533565 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:44.533571 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.533792 (+ 221us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.534460 (+ 668us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:44.534489 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:44.536057 (+ 1568us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:44.537901 (+ 1844us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:44.537908 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:44.537920 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:44.538168 (+ 248us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:44.538430 (+ 262us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:44.538434 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:44.538435 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:44.538523 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:44.538908 (+ 385us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:44.538914 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:44.539131 (+ 217us) client_negotiation.cc:770] Sending connection context
0429 08:00:44.539362 (+ 231us) client_negotiation.cc:241] Negotiation successful
0429 08:00:44.539663 (+ 301us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:00:44.541713 18888 heartbeater.cc:344] Connected to a master server at 127.12.12.190:33377
I20260429 08:00:44.541983 18888 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:44.542495 18888 heartbeater.cc:507] Master 127.12.12.190:33377 requested a full tablet report, sending...
I20260429 08:00:44.543663 18559 ts_manager.cc:194] Registered new tserver with Master: 66e6f21f2e5e4f0db32183663b921491 (127.12.12.130:39451)
I20260429 08:00:44.544231 18559 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:55585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:44.578279 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:33377
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:45659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:44.677843 18895 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:44.678051 18895 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:44.678150 18895 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:44.681294 18895 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:44.681358 18895 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:44.681485 18895 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:44.685784 18895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:33377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:44.686821 18895 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:44.687635 18895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:44.693943 18900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:44.693982 18901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:44.694093 18895 server_base.cc:1061] running on GCE node
W20260429 08:00:44.694265 18903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:44.694604 18895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:44.695111 18895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:44.696316 18895 hybrid_clock.cc:648] HybridClock initialized: now 1777449644696300 us; error 32 us; skew 500 ppm
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.699096 18895 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:00:44.700155 18895 webserver.cc:492] Webserver started at http://127.12.12.131:44379/ using document root <none> and password file <none>
I20260429 08:00:44.700752 18895 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:44.700855 18895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:44.701069 18895 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:44.702692 18895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "56b53133bbcb4535ad080a7b5555eca6"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "d3843ecbedcc5bcaaa9b76286c6c622e"
server_key_iv: "9200c50789e3c6c87250edbfa16507a3"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.703166 18895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "56b53133bbcb4535ad080a7b5555eca6"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "d3843ecbedcc5bcaaa9b76286c6c622e"
server_key_iv: "9200c50789e3c6c87250edbfa16507a3"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.706434 18895 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260429 08:00:44.708386 18910 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.709381 18895 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:00:44.709513 18895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "56b53133bbcb4535ad080a7b5555eca6"
format_stamp: "Formatted at 2026-04-29 08:00:44 on dist-test-slave-nhqz"
server_key: "d3843ecbedcc5bcaaa9b76286c6c622e"
server_key_iv: "9200c50789e3c6c87250edbfa16507a3"
server_key_version: "encryptionkey@0"
I20260429 08:00:44.709614 18895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:44.719353 18895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:44.721851 18895 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:44.722043 18895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:44.722524 18895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:44.723382 18895 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:44.723452 18895 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.723517 18895 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:44.723560 18895 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:44.731962 18895 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:33055
I20260429 08:00:44.731986 19023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:33055 every 8 connection(s)
I20260429 08:00:44.732899 18895 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:44.734143 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 18895
I20260429 08:00:44.734261 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:44.734591 12338 external_mini_cluster.cc:1468] Setting key f9ae14e1c7e671e080b15c0246464804
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:00:44.744791 18598 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.734683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:33687 (local address 127.12.12.190:33377)
0429 08:00:44.734826 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.734829 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.735553 (+ 724us) server_negotiation.cc:408] Connection header received
0429 08:00:44.736439 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.736443 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.736496 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.736580 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.738033 (+ 1453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.738536 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.739203 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.739381 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.741881 (+ 2500us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.741897 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.741899 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.741923 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.743161 (+ 1238us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.743649 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.743651 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.743652 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.743692 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.743932 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.743934 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.743935 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.744062 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.744133 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.744584 (+ 451us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.744672 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 08:00:44.745491 19026 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.734993 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33377 (local address 127.12.12.131:33687)
0429 08:00:44.735446 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:44.735485 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:44.736250 (+ 765us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:44.736725 (+ 475us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:44.736733 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:44.737098 (+ 365us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:44.737893 (+ 795us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:44.737904 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.738688 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.738694 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:44.739081 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:44.739086 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.739292 (+ 206us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.739944 (+ 652us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:44.739971 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:44.741751 (+ 1780us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:44.743310 (+ 1559us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:44.743315 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:44.743324 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:44.743556 (+ 232us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:44.743771 (+ 215us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:44.743773 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:44.743775 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:44.743859 (+ 84us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:44.744146 (+ 287us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:44.744150 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:44.744381 (+ 231us) client_negotiation.cc:770] Sending connection context
0429 08:00:44.744543 (+ 162us) client_negotiation.cc:241] Negotiation successful
0429 08:00:44.744716 (+ 173us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20260429 08:00:44.746531 19024 heartbeater.cc:344] Connected to a master server at 127.12.12.190:33377
I20260429 08:00:44.746805 19024 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:44.747431 19024 heartbeater.cc:507] Master 127.12.12.190:33377 requested a full tablet report, sending...
I20260429 08:00:44.748447 18559 ts_manager.cc:194] Registered new tserver with Master: 56b53133bbcb4535ad080a7b5555eca6 (127.12.12.131:33055)
I20260429 08:00:44.748998 18559 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:33687
I20260429 08:00:44.759866 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/test-admin.keytab.
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.774581 12338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260429 08:00:44.777451 19031 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:44.776044 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.12.12.129:42059)
0429 08:00:44.776360 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.776366 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.776390 (+ 24us) server_negotiation.cc:408] Connection header received
0429 08:00:44.776463 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.776467 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.776624 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.776765 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.777292 (+ 527us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49932: BlockingRecv error: recv got EOF from 127.0.0.1:49932 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":98,"threads_started":1}
W20260429 08:00:44.777704 12338 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc)
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.778771 12338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.779666 12338 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 08:00:44.788205 19031 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.780538 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.12.12.129:42059)
0429 08:00:44.780699 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.780702 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.780752 (+ 50us) server_negotiation.cc:408] Connection header received
0429 08:00:44.780915 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.780917 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.780959 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.781023 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.781612 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.782286 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.782860 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.783080 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.784962 (+ 1882us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.785012 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.785023 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.785046 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.786710 (+ 1664us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.787189 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.787194 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.787195 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.787236 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.787428 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.787430 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.787431 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.787599 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.787739 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.787895 (+ 156us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.788062 (+ 167us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260429 08:00:44.791944 19031 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:44.790911 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.12.12.129:42059)
0429 08:00:44.791037 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.791041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.791149 (+ 108us) server_negotiation.cc:408] Connection header received
0429 08:00:44.791314 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.791337 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.791376 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.791457 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.791838 (+ 381us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49960: BlockingRecv error: recv got EOF from 127.0.0.1:49960 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260429 08:00:44.792124 12338 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc' of type FILE
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.793187 12338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.794059 12338 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 08:00:44.802842 19031 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.794826 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.12.12.129:42059)
0429 08:00:44.794947 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.794951 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.795060 (+ 109us) server_negotiation.cc:408] Connection header received
0429 08:00:44.795185 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.795190 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.795226 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.795305 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.795846 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.796313 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.796950 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.797188 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.799189 (+ 2001us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.799225 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.799229 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.799260 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.801164 (+ 1904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.801657 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.801663 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.801666 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.801721 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.801990 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.801997 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.802000 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.802153 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.802307 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.802528 (+ 221us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.802655 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260429 08:00:44.805977 19031 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:44.805050 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.12.12.129:42059)
0429 08:00:44.805267 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.805272 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.805288 (+ 16us) server_negotiation.cc:408] Connection header received
0429 08:00:44.805338 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.805342 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.805392 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.805493 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.805871 (+ 378us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49982: BlockingRecv error: recv got EOF from 127.0.0.1:49982 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260429 08:00:44.806304 12338 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777449587217345-12338-0/krb5kdc/krb5cc)
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.807353 12338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:44.808210 12338 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[18511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.129@KRBTEST.COM
I20260429 08:00:44.816012 19031 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:44.809006 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.12.12.129:42059)
0429 08:00:44.809136 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:44.809140 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:44.809250 (+ 110us) server_negotiation.cc:408] Connection header received
0429 08:00:44.809416 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:44.809420 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:44.809463 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:44.809542 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:44.810117 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.810585 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:44.811166 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:44.811365 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:44.813309 (+ 1944us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:44.813331 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:44.813333 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:44.813355 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:44.814794 (+ 1439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.815145 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.815147 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.815149 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.815184 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:44.815375 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:44.815377 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:44.815378 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:44.815482 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:44.815574 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:44.815734 (+ 160us) server_negotiation.cc:300] Negotiation successful
0429 08:00:44.815854 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 08:00:44.817011 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18623
I20260429 08:00:44.823802 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18759
I20260429 08:00:44.829514 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18895
I20260429 08:00:44.834748 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 18527
2026-04-29T08:00:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3134 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[19044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[19044](info): set up 2 sockets
Apr 29 08:00:44 dist-test-slave-nhqz krb5kdc[19044](info): commencing operation
krb5kdc: starting...
W20260429 08:00:46.802008 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.945s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:46 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:46.935228 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34475
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:47.036099 19060 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:47.036386 19060 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:47.036484 19060 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:47.039813 19060 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:47.039916 19060 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:47.039963 19060 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:47.040019 19060 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:47.040064 19060 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:47.044435 19060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:47.045580 19060 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:47.046450 19060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:47.051892 19065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:47.051891 19066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.052000 19060 server_base.cc:1061] running on GCE node
W20260429 08:00:47.051891 19068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.052572 19060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:47.053483 19060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:47.054666 19060 hybrid_clock.cc:648] HybridClock initialized: now 1777449647054648 us; error 41 us; skew 500 ppm
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:47.057555 19060 init.cc:377] Logged in from keytab as oryx/127.12.12.190@KRBTEST.COM (short username oryx)
I20260429 08:00:47.058588 19060 webserver.cc:492] Webserver started at http://127.12.12.190:45185/ using document root <none> and password file <none>
I20260429 08:00:47.059084 19060 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:47.059129 19060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:47.059312 19060 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:47.060976 19060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "41490c9b9bf040a1acee45269d08c82d"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "121ea2ac1fc565aad5d526a2e26b8b8e"
server_key_iv: "b7df8dad8893c21f6d6e3e38bbc80cc3"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.061444 19060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "41490c9b9bf040a1acee45269d08c82d"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "121ea2ac1fc565aad5d526a2e26b8b8e"
server_key_iv: "b7df8dad8893c21f6d6e3e38bbc80cc3"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.064564 19060 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:00:47.066663 19075 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.067624 19060 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:47.067755 19060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "41490c9b9bf040a1acee45269d08c82d"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "121ea2ac1fc565aad5d526a2e26b8b8e"
server_key_iv: "b7df8dad8893c21f6d6e3e38bbc80cc3"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.067852 19060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:47.092437 19060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:47.098531 19060 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:47.098749 19060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:47.105479 19060 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:34475
I20260429 08:00:47.105479 19127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:34475 every 8 connection(s)
I20260429 08:00:47.106401 19060 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:47.108997 19128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:47.111032 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19060
I20260429 08:00:47.111160 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:47.111446 12338 external_mini_cluster.cc:1468] Setting key 3834888635ef4f80ffff0c88c841a1a4
I20260429 08:00:47.115425 19128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Bootstrap starting.
I20260429 08:00:47.117825 19128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:47.118516 19128 log.cc:826] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Log is configured to *not* fsync() on all Append() calls
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:47.120491 19128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: No bootstrap required, opened a new log
I20260429 08:00:47.123575 19128 raft_consensus.cc:359] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } }
I20260429 08:00:47.123775 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.112660 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.12.12.190:34475)
0429 08:00:47.113054 (+ 394us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.113064 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.113100 (+ 36us) server_negotiation.cc:408] Connection header received
0429 08:00:47.113742 (+ 642us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.113764 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.114108 (+ 344us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.114440 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.115235 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.116391 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.117113 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.117401 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.119512 (+ 2111us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:47.119532 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:47.119545 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:47.119574 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:47.121452 (+ 1878us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.121807 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.121812 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.121818 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.121893 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.122082 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.122085 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.122087 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.122437 (+ 350us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:47.122612 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:47.122823 (+ 211us) server_negotiation.cc:300] Negotiation successful
0429 08:00:47.123040 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":147,"threads_started":1}
I20260429 08:00:47.123785 19128 raft_consensus.cc:385] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:47.123863 19128 raft_consensus.cc:740] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41490c9b9bf040a1acee45269d08c82d, State: Initialized, Role: FOLLOWER
I20260429 08:00:47.124279 19128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } }
I20260429 08:00:47.124431 19128 raft_consensus.cc:399] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:47.124490 19128 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:47.124626 19128 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:47.125727 19128 raft_consensus.cc:515] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } }
I20260429 08:00:47.126080 19128 leader_election.cc:304] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41490c9b9bf040a1acee45269d08c82d; no voters:
I20260429 08:00:47.126374 19128 leader_election.cc:290] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:47.126467 19133 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:47.126641 19133 raft_consensus.cc:697] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 LEADER]: Becoming Leader. State: Replica: 41490c9b9bf040a1acee45269d08c82d, State: Running, Role: LEADER
I20260429 08:00:47.126919 19133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } }
I20260429 08:00:47.127507 19128 sys_catalog.cc:565] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:47.128429 19134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } }
I20260429 08:00:47.128656 19134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:47.128464 19135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41490c9b9bf040a1acee45269d08c82d. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } }
I20260429 08:00:47.128968 19135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:47.129034 19141 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:47.131671 19141 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:47.136292 19141 catalog_manager.cc:1357] Generated new cluster ID: 8aab4ba730d34e47b2f51a81e40442ca
I20260429 08:00:47.136360 19141 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:47.147423 19141 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:47.148288 19141 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:47.156281 19141 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Generated new TSK 0
I20260429 08:00:47.156878 19141 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:47.205533 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34475
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:47.303594 19156 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:47.303829 19156 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:47.303915 19156 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:47.307042 19156 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:47.307134 19156 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:47.307281 19156 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:47.311293 19156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:47.312297 19156 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:47.313134 19156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:47.319136 19162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:47.319140 19164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.319301 19156 server_base.cc:1061] running on GCE node
W20260429 08:00:47.319388 19161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.319747 19156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:47.320276 19156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:47.321435 19156 hybrid_clock.cc:648] HybridClock initialized: now 1777449647321422 us; error 30 us; skew 500 ppm
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:47.323957 19156 init.cc:377] Logged in from keytab as oryx/127.12.12.129@KRBTEST.COM (short username oryx)
I20260429 08:00:47.324970 19156 webserver.cc:492] Webserver started at http://127.12.12.129:37317/ using document root <none> and password file <none>
I20260429 08:00:47.325446 19156 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:47.325489 19156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:47.325637 19156 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:47.327168 19156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "fad2a529f4a346358315acf2866329d1"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "194768745bef44922c4118808eadc729"
server_key_iv: "01cc7a0710ba581dcb57268d4acea97d"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.327585 19156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "fad2a529f4a346358315acf2866329d1"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "194768745bef44922c4118808eadc729"
server_key_iv: "01cc7a0710ba581dcb57268d4acea97d"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.330771 19156 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260429 08:00:47.332626 19171 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.333554 19156 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:47.333662 19156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "fad2a529f4a346358315acf2866329d1"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "194768745bef44922c4118808eadc729"
server_key_iv: "01cc7a0710ba581dcb57268d4acea97d"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.333750 19156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:47.343637 19156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:47.346328 19156 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:47.346496 19156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:47.347024 19156 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:47.347852 19156 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:47.347898 19156 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.347962 19156 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:47.347991 19156 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.356966 19156 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:36893
I20260429 08:00:47.357000 19284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:36893 every 8 connection(s)
I20260429 08:00:47.357821 19156 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:47.360843 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19156
I20260429 08:00:47.360965 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:47.361275 12338 external_mini_cluster.cc:1468] Setting key 336d425e71c56eb8066b32aaa487ed03
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:47.370440 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.359371 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:45875 (local address 127.12.12.190:34475)
0429 08:00:47.359502 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.359505 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.360103 (+ 598us) server_negotiation.cc:408] Connection header received
0429 08:00:47.361042 (+ 939us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.361047 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.361103 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.361178 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.362527 (+ 1349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.363017 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.363677 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.363862 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.366757 (+ 2895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:47.366780 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:47.366785 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:47.366818 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:47.368501 (+ 1683us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.369002 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.369008 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.369013 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.369073 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.369346 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.369352 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.369355 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.369562 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:47.369659 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:47.370127 (+ 468us) server_negotiation.cc:300] Negotiation successful
0429 08:00:47.370261 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:00:47.370934 19287 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.359612 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:45875)
0429 08:00:47.359985 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:47.360015 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:47.360805 (+ 790us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:47.361357 (+ 552us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:47.361369 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:47.361818 (+ 449us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:47.362383 (+ 565us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.362398 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.363165 (+ 767us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.363170 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:47.363561 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.363567 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.363761 (+ 194us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.364909 (+ 1148us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:47.364936 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:47.366608 (+ 1672us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:47.368623 (+ 2015us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.368628 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.368636 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.368906 (+ 270us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.369172 (+ 266us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.369174 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.369176 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.369251 (+ 75us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.369665 (+ 414us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:47.369668 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:47.369873 (+ 205us) client_negotiation.cc:770] Sending connection context
0429 08:00:47.370016 (+ 143us) client_negotiation.cc:241] Negotiation successful
0429 08:00:47.370178 (+ 162us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260429 08:00:47.372046 19285 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34475
I20260429 08:00:47.372268 19285 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:47.372767 19285 heartbeater.cc:507] Master 127.12.12.190:34475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.130@KRBTEST.COM" created.
I20260429 08:00:47.374140 19092 ts_manager.cc:194] Registered new tserver with Master: fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893)
I20260429 08:00:47.375576 19092 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:45875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:47.409794 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34475
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:47.508438 19292 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:47.508687 19292 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:47.508767 19292 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:47.511771 19292 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:47.511852 19292 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:47.511952 19292 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:47.516032 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:47.517061 19292 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:47.517884 19292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:47.524067 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:47.524101 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:47.524067 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.524420 19292 server_base.cc:1061] running on GCE node
I20260429 08:00:47.524746 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:47.525313 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:47.526453 19292 hybrid_clock.cc:648] HybridClock initialized: now 1777449647526433 us; error 32 us; skew 500 ppm
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:47.529098 19292 init.cc:377] Logged in from keytab as oryx/127.12.12.130@KRBTEST.COM (short username oryx)
I20260429 08:00:47.530027 19292 webserver.cc:492] Webserver started at http://127.12.12.130:39435/ using document root <none> and password file <none>
I20260429 08:00:47.530543 19292 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:47.530601 19292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:47.530741 19292 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:47.532159 19292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "f597d7c0ffe64506a90069b5bf96ce88"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "95283d67093c249fd34bdd819bdf4bf8"
server_key_iv: "f537f8c35c85a261196430665ee30688"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.532593 19292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "f597d7c0ffe64506a90069b5bf96ce88"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "95283d67093c249fd34bdd819bdf4bf8"
server_key_iv: "f537f8c35c85a261196430665ee30688"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.535542 19292 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:47.537451 19307 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.538403 19292 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:47.538511 19292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "f597d7c0ffe64506a90069b5bf96ce88"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "95283d67093c249fd34bdd819bdf4bf8"
server_key_iv: "f537f8c35c85a261196430665ee30688"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.538601 19292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:47.547400 19292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:47.549980 19292 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:47.550159 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:47.550643 19292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:47.551404 19292 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:47.551448 19292 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.551508 19292 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:47.551563 19292 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.560405 19292 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:39375
I20260429 08:00:47.560442 19420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:39375 every 8 connection(s)
I20260429 08:00:47.561306 19292 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:47.564910 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19292
I20260429 08:00:47.565022 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:47.565267 12338 external_mini_cluster.cc:1468] Setting key bf02174d23160eb5f961f7abb1f561d2
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:00:47.573943 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.562978 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:42321 (local address 127.12.12.190:34475)
0429 08:00:47.563163 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.563167 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.563756 (+ 589us) server_negotiation.cc:408] Connection header received
0429 08:00:47.564573 (+ 817us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.564577 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.564658 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.564755 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.566157 (+ 1402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.566801 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.567424 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.567598 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.570452 (+ 2854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:47.570472 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:47.570475 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:47.570499 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:47.572080 (+ 1581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.572563 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.572568 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.572572 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.572623 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.572962 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.572966 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.572968 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.573108 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:47.573207 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:47.573670 (+ 463us) server_negotiation.cc:300] Negotiation successful
0429 08:00:47.573784 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 08:00:47.574561 19423 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.563234 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:42321)
0429 08:00:47.563622 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:47.563653 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:47.564383 (+ 730us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:47.564930 (+ 547us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:47.564937 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:47.565418 (+ 481us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:47.566023 (+ 605us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.566037 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.566931 (+ 894us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.566935 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:47.567313 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.567319 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.567513 (+ 194us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.568663 (+ 1150us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:47.568706 (+ 43us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:47.570299 (+ 1593us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:47.572192 (+ 1893us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.572198 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.572210 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.572454 (+ 244us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.572722 (+ 268us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.572725 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.572727 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.572852 (+ 125us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.573208 (+ 356us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:47.573215 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:47.573432 (+ 217us) client_negotiation.cc:770] Sending connection context
0429 08:00:47.573622 (+ 190us) client_negotiation.cc:241] Negotiation successful
0429 08:00:47.573845 (+ 223us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260429 08:00:47.575670 19421 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34475
I20260429 08:00:47.575949 19421 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.131@KRBTEST.COM" created.
I20260429 08:00:47.576555 19421 heartbeater.cc:507] Master 127.12.12.190:34475 requested a full tablet report, sending...
I20260429 08:00:47.577688 19092 ts_manager.cc:194] Registered new tserver with Master: f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375)
I20260429 08:00:47.578236 19092 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:42321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:47.611876 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34475
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:47.708665 19428 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:47.708868 19428 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:47.708920 19428 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:47.711988 19428 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:47.712040 19428 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:47.712170 19428 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:47.716150 19428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:47.717028 19428 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:47.717778 19428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:47.723726 19436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:47.723731 19433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.723956 19428 server_base.cc:1061] running on GCE node
W20260429 08:00:47.723742 19434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.724356 19428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:47.724906 19428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:47.726048 19428 hybrid_clock.cc:648] HybridClock initialized: now 1777449647726033 us; error 27 us; skew 500 ppm
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:47.728504 19428 init.cc:377] Logged in from keytab as oryx/127.12.12.131@KRBTEST.COM (short username oryx)
I20260429 08:00:47.729516 19428 webserver.cc:492] Webserver started at http://127.12.12.131:42983/ using document root <none> and password file <none>
I20260429 08:00:47.730047 19428 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:47.730113 19428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:47.730314 19428 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:47.731797 19428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "57838a88843949328cd35e764aa23bb8"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "9d6f7a6b4a30a11d59e94af1f98e6404"
server_key_iv: "0a0184282455a5d7024ab7e7f14bdc22"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.732239 19428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "57838a88843949328cd35e764aa23bb8"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "9d6f7a6b4a30a11d59e94af1f98e6404"
server_key_iv: "0a0184282455a5d7024ab7e7f14bdc22"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.735219 19428 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:47.737136 19443 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.737968 19428 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:47.738088 19428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "57838a88843949328cd35e764aa23bb8"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "9d6f7a6b4a30a11d59e94af1f98e6404"
server_key_iv: "0a0184282455a5d7024ab7e7f14bdc22"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.738176 19428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:47.749147 19428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:47.751241 19428 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:47.751438 19428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:47.751912 19428 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:47.752687 19428 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:47.752753 19428 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.752815 19428 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:47.752902 19428 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.761704 19428 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:35323
I20260429 08:00:47.761729 19556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:35323 every 8 connection(s)
I20260429 08:00:47.762543 19428 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:47.766366 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19428
I20260429 08:00:47.766480 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:47.766731 12338 external_mini_cluster.cc:1468] Setting key b7455041601a8b3773c360dbd3a44e2e
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:47.774344 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.764047 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:44223 (local address 127.12.12.190:34475)
0429 08:00:47.764229 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.764234 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.764834 (+ 600us) server_negotiation.cc:408] Connection header received
0429 08:00:47.765613 (+ 779us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.765616 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.765664 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.765737 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.767079 (+ 1342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.767772 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.768435 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.768608 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.770992 (+ 2384us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:47.771007 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:47.771009 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:47.771034 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:47.772565 (+ 1531us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.773113 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.773118 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.773124 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.773183 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.773493 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.773496 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.773497 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.773632 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:47.773718 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:47.774127 (+ 409us) server_negotiation.cc:300] Negotiation successful
0429 08:00:47.774215 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260429 08:00:47.774974 19559 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.764329 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:44223)
0429 08:00:47.764697 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:47.764726 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:47.765450 (+ 724us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:47.765869 (+ 419us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:47.765876 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:47.766300 (+ 424us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:47.766915 (+ 615us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.766931 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.767895 (+ 964us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.767900 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:47.768312 (+ 412us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.768319 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.768507 (+ 188us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.769484 (+ 977us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:47.769500 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:47.770864 (+ 1364us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:47.772675 (+ 1811us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.772681 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.772692 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.773016 (+ 324us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.773311 (+ 295us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.773314 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.773315 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.773402 (+ 87us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.773725 (+ 323us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:47.773729 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:47.773907 (+ 178us) client_negotiation.cc:770] Sending connection context
0429 08:00:47.774057 (+ 150us) client_negotiation.cc:241] Negotiation successful
0429 08:00:47.774244 (+ 187us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
I20260429 08:00:47.775933 19557 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34475
I20260429 08:00:47.776156 19557 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:47.776625 19557 heartbeater.cc:507] Master 127.12.12.190:34475 requested a full tablet report, sending...
I20260429 08:00:47.777578 19092 ts_manager.cc:194] Registered new tserver with Master: 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323)
I20260429 08:00:47.778097 19092 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:44223
I20260429 08:00:47.780186 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:47.824736 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.189:45059
--webserver_interface=127.12.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34475,127.12.12.189:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:47.922116 19564 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:47.922348 19564 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:47.922439 19564 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:47.925752 19564 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:47.925822 19564 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:47.925863 19564 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:47.925925 19564 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:47.925962 19564 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:47.930230 19564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34475,127.12.12.189:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.189:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.12.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
W20260429 08:00:47.930447 19564 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.12.190:34475,127.12.12.189:45059'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260429 08:00:47.931252 19564 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:47.932031 19564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:47.937016 19569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:47.937059 19572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.937212 19564 server_base.cc:1061] running on GCE node
W20260429 08:00:47.937317 19570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:47.937793 19564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:47.938593 19564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:47.939778 19564 hybrid_clock.cc:648] HybridClock initialized: now 1777449647939761 us; error 37 us; skew 500 ppm
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:47.942493 19564 init.cc:377] Logged in from keytab as oryx/127.12.12.189@KRBTEST.COM (short username oryx)
I20260429 08:00:47.943449 19564 webserver.cc:492] Webserver started at http://127.12.12.189:42207/ using document root <none> and password file <none>
I20260429 08:00:47.943918 19564 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:47.943965 19564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:47.944113 19564 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:47.945709 19564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/instance:
uuid: "177399518fce4172b5943114d5d36fb5"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "fec8f0d9df14d76a0226a6f7fe6bc368"
server_key_iv: "c675f1e938035186e1660fec7877b428"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.946115 19564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal/instance:
uuid: "177399518fce4172b5943114d5d36fb5"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "fec8f0d9df14d76a0226a6f7fe6bc368"
server_key_iv: "c675f1e938035186e1660fec7877b428"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.949216 19564 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:00:47.951164 19579 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:47.952050 19564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:47.952183 19564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
uuid: "177399518fce4172b5943114d5d36fb5"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "fec8f0d9df14d76a0226a6f7fe6bc368"
server_key_iv: "c675f1e938035186e1660fec7877b428"
server_key_version: "encryptionkey@0"
I20260429 08:00:47.952281 19564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:47.967604 19564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:47.970181 19564 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:47.970382 19564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:47.977586 19564 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.189:45059
I20260429 08:00:47.977592 19631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.189:45059 every 8 connection(s)
I20260429 08:00:47.978564 19564 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/info.pb
I20260429 08:00:47.980147 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19564
I20260429 08:00:47.981438 19632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:47.985755 19632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.189@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:47.993676 19635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.981502 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.12.12.189:45059)
0429 08:00:47.981892 (+ 390us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.981903 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.981942 (+ 39us) server_negotiation.cc:408] Connection header received
0429 08:00:47.982725 (+ 783us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.982769 (+ 44us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.983126 (+ 357us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.983476 (+ 350us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.984241 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.985459 (+ 1218us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.986079 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.986377 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.988795 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:47.988902 (+ 107us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:47.988909 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:47.988935 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:47.991109 (+ 2174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.991510 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.991516 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.991524 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.991605 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.991840 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.991842 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.991843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.992127 (+ 284us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:47.992287 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:47.992559 (+ 272us) server_negotiation.cc:300] Negotiation successful
0429 08:00:47.992782 (+ 223us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260429 08:00:47.995652 19636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.987339 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35180)
0429 08:00:47.987685 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:47.987714 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:47.987838 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:47.988142 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:47.988147 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:47.988573 (+ 426us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:47.988792 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.988798 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.989755 (+ 957us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.989758 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:47.990375 (+ 617us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.990384 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.990497 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.990902 (+ 405us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:47.990922 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:47.992511 (+ 1589us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:47.994274 (+ 1763us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.994281 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.994284 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.994484 (+ 200us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.994750 (+ 266us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:47.994753 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:47.994754 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:47.994796 (+ 42us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:47.995155 (+ 359us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:47.995162 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:47.995256 (+ 94us) client_negotiation.cc:770] Sending connection context
0429 08:00:47.995405 (+ 149us) client_negotiation.cc:241] Negotiation successful
0429 08:00:47.995530 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":135,"threads_started":1}
I20260429 08:00:47.995779 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.987396 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.12.12.190:34475)
0429 08:00:47.987554 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.987570 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.987786 (+ 216us) server_negotiation.cc:408] Connection header received
0429 08:00:47.987944 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.987947 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.988001 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.988084 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.988959 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.989614 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.990489 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.990626 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.992634 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:47.992653 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:47.992656 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:47.992683 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:47.994173 (+ 1490us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.994598 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.994605 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.994607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.994650 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:47.994886 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:47.994889 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:47.994891 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:47.995053 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:47.995156 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:47.995555 (+ 399us) server_negotiation.cc:300] Negotiation successful
0429 08:00:47.995648 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 08:00:47.996681 19632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 08:00:47 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.189@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
I20260429 08:00:48.002753 19636 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.997152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49098)
0429 08:00:47.997265 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:47.997279 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:47.997346 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:47.997520 (+ 174us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:47.997522 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:47.997654 (+ 132us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:47.997823 (+ 169us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.997827 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.998469 (+ 642us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.998471 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:47.998797 (+ 326us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:47.998801 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.998879 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:47.999268 (+ 389us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:47.999280 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.000346 (+ 1066us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.001799 (+ 1453us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.001802 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.001803 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.001948 (+ 145us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.002128 (+ 180us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.002130 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.002131 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.002167 (+ 36us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.002413 (+ 246us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.002415 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.002472 (+ 57us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.002544 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.002647 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260429 08:00:48.002836 19635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:47.997225 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.12.12.189:45059)
0429 08:00:47.997337 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:47.997340 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:47.997351 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:47.997403 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:47.997406 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:47.997438 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:47.997499 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:47.997928 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.998389 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:47.998876 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:47.998996 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.000442 (+ 1446us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.000454 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.000456 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.000474 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.001709 (+ 1235us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.002026 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.002028 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.002029 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.002062 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.002229 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.002230 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.002231 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.002345 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.002413 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.002630 (+ 217us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.002712 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 08:00:48.005448 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Bootstrap starting.
I20260429 08:00:48.007661 19632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:48.008351 19632 log.cc:826] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:48.010079 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: No bootstrap required, opened a new log
I20260429 08:00:48.012394 19632 raft_consensus.cc:359] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } }
I20260429 08:00:48.012579 19632 raft_consensus.cc:385] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:48.012696 19632 raft_consensus.cc:740] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177399518fce4172b5943114d5d36fb5, State: Initialized, Role: FOLLOWER
I20260429 08:00:48.013167 19632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } }
I20260429 08:00:48.013573 19638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } } }
I20260429 08:00:48.013677 19638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:48.014150 19632 sys_catalog.cc:565] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: configured and running, proceeding with master startup.
W20260429 08:00:48.018764 19649 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 08:00:48.018868 19649 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 08:00:48.020992 19564 master_runner.cc:428] Detected that this master 177399518fce4172b5943114d5d36fb5 is joining an existing cluster
I20260429 08:00:48.021059 19564 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.12.189:45059
I20260429 08:00:48.030061 19635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.023187 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.12.12.189:45059)
0429 08:00:48.023304 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.023308 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.024024 (+ 716us) server_negotiation.cc:408] Connection header received
0429 08:00:48.024147 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.024153 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.024191 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.024236 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.024911 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.025342 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.025996 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.026205 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.026931 (+ 726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.026947 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.026950 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.026974 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.028681 (+ 1707us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.029071 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.029074 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.029076 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.029123 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.029353 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.029356 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.029358 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.029489 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.029608 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.029832 (+ 224us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.029947 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":40}
I20260429 08:00:48.030071 19656 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.023047 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49114)
0429 08:00:48.023955 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.023965 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.024057 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.024350 (+ 293us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.024352 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.024524 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.024760 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.024768 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.025433 (+ 665us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.025437 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.025874 (+ 437us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.025881 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.025982 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.026546 (+ 564us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.026557 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.026823 (+ 266us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.028774 (+ 1951us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.028777 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.028778 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.028983 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.029204 (+ 221us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.029206 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.029208 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.029249 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.029581 (+ 332us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.029583 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.029668 (+ 85us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.029783 (+ 115us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.029897 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":112,"threads_started":1}
I20260429 08:00:48.040571 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.023131 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.12.12.190:34475)
0429 08:00:48.023255 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.023260 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.023769 (+ 509us) server_negotiation.cc:408] Connection header received
0429 08:00:48.023916 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.023920 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.023974 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.024058 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.024784 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.025596 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.026400 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.026614 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.037903 (+ 11289us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.037921 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.037923 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.037948 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.039238 (+ 1290us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.039645 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.039648 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.039649 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.039688 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.039890 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.039891 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.039893 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.040022 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.040088 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.040320 (+ 232us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.040398 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:48.040603 19655 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.023047 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35196)
0429 08:00:48.023662 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.023679 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.023813 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.024093 (+ 280us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.024099 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.024350 (+ 251us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.024660 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.024667 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.025741 (+ 1074us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.025745 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.026292 (+ 547us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.026299 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.026408 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.037378 (+ 10970us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.037394 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.037740 (+ 346us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.039339 (+ 1599us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.039342 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.039345 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.039559 (+ 214us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.039764 (+ 205us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.039767 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.039769 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.039818 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.040094 (+ 276us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.040096 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.040178 (+ 82us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.040270 (+ 92us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.040375 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":283,"thread_start_us":110,"threads_started":1}
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.190@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
I20260429 08:00:48.052903 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.044483 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49124)
0429 08:00:48.044838 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.044873 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.045007 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.045312 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.045316 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.045692 (+ 376us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.045906 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.045912 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.046704 (+ 792us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.046710 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.047043 (+ 333us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.047049 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.047161 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.047675 (+ 514us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.047688 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.049181 (+ 1493us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.051163 (+ 1982us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.051175 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.051183 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.051489 (+ 306us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.051773 (+ 284us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.051779 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.051783 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.051852 (+ 69us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.052286 (+ 434us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.052292 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.052401 (+ 109us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.052549 (+ 148us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.052692 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":125,"threads_started":1}
I20260429 08:00:48.052949 19635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.044534 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.12.12.189:45059)
0429 08:00:48.044653 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.044659 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.044941 (+ 282us) server_negotiation.cc:408] Connection header received
0429 08:00:48.045126 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.045129 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.045179 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.045272 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.046016 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.046605 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.047146 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.047346 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.049303 (+ 1957us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.049319 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.049323 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.049348 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.051028 (+ 1680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.051626 (+ 598us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.051630 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.051632 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.051677 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.051953 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.051956 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.051958 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.052135 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.052225 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.052627 (+ 402us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.052730 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 08:00:48.054656 19092 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.12.12.189:45059: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "177399518fce4172b5943114d5d36fb5"
member_type: NON_VOTER
last_known_addr {
host: "127.12.12.189"
port: 45059
}
attrs {
promote: true
}
}
dest_uuid: "41490c9b9bf040a1acee45269d08c82d"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.12.12.188@KRBTEST.COM; defaulting to no policy
I20260429 08:00:48.055356 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } }
I20260429 08:00:48.056895 19660 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 177399518fce4172b5943114d5d36fb5 (127.12.12.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } }
I20260429 08:00:48.057631 19606 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.188@KRBTEST.COM" created.
I20260429 08:00:48.058717 19660 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.12.12.189:45059
I20260429 08:00:48.059001 19606 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Refusing update from remote peer 41490c9b9bf040a1acee45269d08c82d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260429 08:00:48.059088 19660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } }
I20260429 08:00:48.059209 19660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:48.059482 19564 master.cc:562] Master@127.12.12.189:45059 shutting down...
I20260429 08:00:48.059728 19659 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260429 08:00:48.061479 19078 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260429 08:00:48.063796 19564 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:48.064038 19564 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:48.064134 19564 tablet_replica.cc:333] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: stopping tablet replica
I20260429 08:00:48.068481 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.068140 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49136)
0429 08:00:48.068353 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.068407 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.189:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260429 08:00:48.068651 19078 consensus_peers.cc:597] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d -> Peer 177399518fce4172b5943114d5d36fb5 (127.12.12.189:45059): Couldn't send request to peer 177399518fce4172b5943114d5d36fb5. Status: Network error: Client connection negotiation failed: client connection to 127.12.12.189:45059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260429 08:00:48.069271 19564 master.cc:584] Master@127.12.12.189:45059 shutdown complete.
I20260429 08:00:48.069321 19564 master_runner.cc:305] Clearing existing system tablet
I20260429 08:00:48.069536 19564 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:48.069598 19564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:48.070815 19564 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.071688 19665 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:48.072057 19564 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:48.072207 19564 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.072252 19564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
uuid: "177399518fce4172b5943114d5d36fb5"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "fec8f0d9df14d76a0226a6f7fe6bc368"
server_key_iv: "c675f1e938035186e1660fec7877b428"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.073995 19564 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:48.076097 19564 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260429 08:00:48.076213 19564 log.cc:1199] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260429 08:00:48.076509 19564 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Deleting consensus metadata
I20260429 08:00:48.076753 19564 master_runner.cc:315] Copying system tablet from 127.12.12.190:34475
I20260429 08:00:48.077891 19564 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: tablet copy: Beginning tablet copy session from remote peer at address 127.12.12.190:34475
WARNING: no policy specified for HTTP/127.12.12.188@KRBTEST.COM; defaulting to no policy
I20260429 08:00:48.084453 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.078400 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.12.12.190:34475)
0429 08:00:48.078518 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.078522 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.078726 (+ 204us) server_negotiation.cc:408] Connection header received
0429 08:00:48.078871 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.078874 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.078925 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.079003 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.079504 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.079978 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.080619 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.080868 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.081646 (+ 778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.081667 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.081670 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.081689 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.082995 (+ 1306us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.083406 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.083410 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.083411 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.083457 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.083695 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.083698 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.083700 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.083874 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.083987 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.084216 (+ 229us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.084320 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 08:00:48.084466 19672 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.078361 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35210)
0429 08:00:48.078644 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.078656 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.078771 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.079018 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.079021 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.079204 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.079391 (+ 187us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.079396 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.080081 (+ 685us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.080086 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.080499 (+ 413us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.080506 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.080634 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.081218 (+ 584us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.081231 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.081525 (+ 294us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.083109 (+ 1584us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.083113 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.083115 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.083302 (+ 187us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.083558 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.083561 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.083562 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.083605 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.083976 (+ 371us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.083979 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.084062 (+ 83us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.084165 (+ 103us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.084290 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.188@KRBTEST.COM" created.
I20260429 08:00:48.085152 19112 tablet_copy_service.cc:140] P 41490c9b9bf040a1acee45269d08c82d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 177399518fce4172b5943114d5d36fb5 ({username='oryx', principal='oryx/127.12.12.189@KRBTEST.COM'} at 127.0.0.1:35210)
I20260429 08:00:48.085335 19112 tablet_copy_service.cc:161] P 41490c9b9bf040a1acee45269d08c82d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 177399518fce4172b5943114d5d36fb5 at {username='oryx', principal='oryx/127.12.12.189@KRBTEST.COM'} at 127.0.0.1:35210: session id = 177399518fce4172b5943114d5d36fb5-00000000000000000000000000000000
I20260429 08:00:48.087164 19112 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Tablet Copy: opened 0 blocks and 1 log segments
I20260429 08:00:48.088689 19564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260429 08:00:48.090325 19564 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: tablet copy: Starting download of 0 data blocks...
I20260429 08:00:48.090502 19564 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: tablet copy: Starting download of 1 WAL segments...
I20260429 08:00:48.092165 19564 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260429 08:00:48.093739 19112 tablet_copy_service.cc:342] P 41490c9b9bf040a1acee45269d08c82d: Request end of tablet copy session 177399518fce4172b5943114d5d36fb5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.12.189@KRBTEST.COM'} at 127.0.0.1:35210
I20260429 08:00:48.093858 19112 tablet_copy_service.cc:434] P 41490c9b9bf040a1acee45269d08c82d: ending tablet copy session 177399518fce4172b5943114d5d36fb5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 177399518fce4172b5943114d5d36fb5
W20260429 08:00:48.098606 19564 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:48.100322 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.188:40281
--webserver_interface=127.12.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 08:00:48.100952 19564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:48.102730 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:48.102754 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:48.103288 19564 server_base.cc:1061] running on GCE node
W20260429 08:00:48.109076 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:48.109381 19564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:48.109671 19564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:48.110817 19564 hybrid_clock.cc:648] HybridClock initialized: now 1777449648110804 us; error 29 us; skew 500 ppm
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449648, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:48.112520 19564 init.cc:377] Logged in from keytab as oryx/127.12.12.189@KRBTEST.COM (short username oryx)
I20260429 08:00:48.113039 19564 webserver.cc:492] Webserver started at http://127.12.12.189:44161/ using document root <none> and password file <none>
I20260429 08:00:48.113211 19564 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:48.113241 19564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:48.114820 19564 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.115830 19685 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:48.116097 19564 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.116160 19564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
uuid: "177399518fce4172b5943114d5d36fb5"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "fec8f0d9df14d76a0226a6f7fe6bc368"
server_key_iv: "c675f1e938035186e1660fec7877b428"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.116238 19564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:48.159564 19564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:48.161729 19564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:48.166651 19564 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.189:45059
I20260429 08:00:48.166718 19737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.189:45059 every 8 connection(s)
I20260429 08:00:48.167357 19564 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-1/data/info.pb
I20260429 08:00:48.169202 19738 sys_catalog.cc:263] Verifying existing consensus state
I20260429 08:00:48.170284 19738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Bootstrap starting.
I20260429 08:00:48.180480 19738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 08:00:48.180830 19738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: Bootstrap complete.
I20260429 08:00:48.181480 19738 raft_consensus.cc:359] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } }
I20260429 08:00:48.181649 19738 raft_consensus.cc:740] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 177399518fce4172b5943114d5d36fb5, State: Initialized, Role: LEARNER
I20260429 08:00:48.181800 19738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } }
I20260429 08:00:48.182143 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } }
I20260429 08:00:48.182274 19738 sys_catalog.cc:565] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:48.182313 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: LEARNER
I20260429 08:00:48.185881 19752 catalog_manager.cc:1269] Loaded cluster ID: 8aab4ba730d34e47b2f51a81e40442ca
I20260429 08:00:48.185941 19752 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: loading cluster ID for follower catalog manager: success
I20260429 08:00:48.188051 19752 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: acquiring CA information for follower catalog manager: success
I20260429 08:00:48.189239 19752 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260429 08:00:48.221218 19674 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:48.221447 19674 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:48.221537 19674 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:48.224922 19674 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:48.224987 19674 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:48.225032 19674 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:48.225067 19674 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:48.225106 19674 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:48.229463 19674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.188:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.12.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:48.230567 19674 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:48.231377 19674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:48.237155 19757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:48.237212 19758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:48.237250 19760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:48.237371 19674 server_base.cc:1061] running on GCE node
I20260429 08:00:48.237932 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:48.238834 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:48.240003 19674 hybrid_clock.cc:648] HybridClock initialized: now 1777449648239996 us; error 41 us; skew 500 ppm
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449648, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:48.242995 19674 init.cc:377] Logged in from keytab as oryx/127.12.12.188@KRBTEST.COM (short username oryx)
I20260429 08:00:48.244009 19674 webserver.cc:492] Webserver started at http://127.12.12.188:38683/ using document root <none> and password file <none>
I20260429 08:00:48.244573 19674 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:48.244647 19674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:48.244902 19674 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:48.246647 19674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/instance:
uuid: "059cb565568c418e80d1a4b2d88a9477"
format_stamp: "Formatted at 2026-04-29 08:00:48 on dist-test-slave-nhqz"
server_key: "b23b8ff0e159f28ddc113c01ed47aee8"
server_key_iv: "42db3db858a2403f175cd33169b913ed"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.247114 19674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal/instance:
uuid: "059cb565568c418e80d1a4b2d88a9477"
format_stamp: "Formatted at 2026-04-29 08:00:48 on dist-test-slave-nhqz"
server_key: "b23b8ff0e159f28ddc113c01ed47aee8"
server_key_iv: "42db3db858a2403f175cd33169b913ed"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.250445 19674 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260429 08:00:48.252435 19767 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:48.253484 19674 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:48.253611 19674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
uuid: "059cb565568c418e80d1a4b2d88a9477"
format_stamp: "Formatted at 2026-04-29 08:00:48 on dist-test-slave-nhqz"
server_key: "b23b8ff0e159f28ddc113c01ed47aee8"
server_key_iv: "42db3db858a2403f175cd33169b913ed"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.253710 19674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:48.295049 19674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:48.298272 19674 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:48.298498 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:48.306031 19674 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.188:40281
I20260429 08:00:48.306033 19819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.188:40281 every 8 connection(s)
I20260429 08:00:48.307068 19674 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/info.pb
I20260429 08:00:48.309865 19820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:48.313331 19820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } has no permanent_uuid. Determining permanent_uuid...
I20260429 08:00:48.317070 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19674
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449648, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.188@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.188@KRBTEST.COM
I20260429 08:00:48.326385 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.314991 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.12.12.190:34475)
0429 08:00:48.315132 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.315135 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.315611 (+ 476us) server_negotiation.cc:408] Connection header received
0429 08:00:48.316288 (+ 677us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.316292 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.316354 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.316455 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.317800 (+ 1345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.318472 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.319101 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.319286 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.322206 (+ 2920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.322228 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.322230 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.322255 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.323813 (+ 1558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.324663 (+ 850us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.324668 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.324672 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.324729 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.325069 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.325072 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.325073 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.325206 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.325294 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.326163 (+ 869us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.326254 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 08:00:48.326836 19822 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.315069 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35214)
0429 08:00:48.315482 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.315512 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.316106 (+ 594us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.316598 (+ 492us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.316605 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.317051 (+ 446us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.317634 (+ 583us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.317653 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.318594 (+ 941us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.318598 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.318986 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.318992 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.319131 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.320381 (+ 1250us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.320398 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.322084 (+ 1686us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.324175 (+ 2091us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.324186 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.324197 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.324535 (+ 338us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.324844 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.324848 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.324850 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.324970 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.325328 (+ 358us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.325334 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.325596 (+ 262us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.325816 (+ 220us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.326009 (+ 193us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260429 08:00:48.327244 19823 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.317890 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.12.12.188:40281)
0429 08:00:48.318193 (+ 303us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.318198 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.318216 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:00:48.318248 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.318251 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.318366 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.318480 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.319444 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.320429 (+ 985us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.321261 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.321490 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.323660 (+ 2170us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.323681 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.323690 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.323710 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.325724 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.326084 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.326089 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.326091 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.326134 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.326403 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.326406 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.326408 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.326601 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.326756 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.326957 (+ 201us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.327092 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":149,"threads_started":1}
I20260429 08:00:48.328168 19820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449648, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.188@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
I20260429 08:00:48.335096 19822 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.328656 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49148)
0429 08:00:48.328847 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.328863 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.328927 (+ 64us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.329307 (+ 380us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.329309 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.329433 (+ 124us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.329582 (+ 149us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.329586 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.330523 (+ 937us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.330526 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.330884 (+ 358us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.330890 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.330969 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.331391 (+ 422us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.331403 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.332475 (+ 1072us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.334106 (+ 1631us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.334109 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.334111 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.334260 (+ 149us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.334452 (+ 192us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.334454 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.334455 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.334493 (+ 38us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.334763 (+ 270us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.334765 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.334827 (+ 62us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.334902 (+ 75us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.334983 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260429 08:00:48.335155 19824 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.328767 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.12.12.189:45059)
0429 08:00:48.329099 (+ 332us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.329103 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.329121 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:00:48.329156 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.329158 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.329200 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.329276 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.329679 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.330431 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.330971 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.331093 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.332629 (+ 1536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.332645 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.332648 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.332670 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.334009 (+ 1339us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.334339 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.334341 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.334343 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.334380 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.334569 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.334572 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.334573 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.334692 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.334763 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.334950 (+ 187us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.335038 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":133,"threads_started":1}
I20260429 08:00:48.335948 19820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449648, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.188@KRBTEST.COM for oryx/127.12.12.188@KRBTEST.COM
I20260429 08:00:48.342633 19822 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.336450 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:35364)
0429 08:00:48.336571 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.336585 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.336654 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.336895 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.336897 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.337013 (+ 116us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.337149 (+ 136us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.337153 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.337822 (+ 669us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.337824 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.338125 (+ 301us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.338130 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.338213 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.338614 (+ 401us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.338626 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.339659 (+ 1033us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.341388 (+ 1729us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.341391 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.341392 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.341553 (+ 161us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.341749 (+ 196us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.341751 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.341752 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.341793 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.342154 (+ 361us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.342158 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.342224 (+ 66us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.342346 (+ 122us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.342458 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260429 08:00:48.342635 19823 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.336516 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.12.12.188:40281)
0429 08:00:48.336629 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.336632 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.336643 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:48.336718 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.336720 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.336756 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.336885 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.337236 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.337723 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.338209 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.338351 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.339779 (+ 1428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.339792 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.339794 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.339813 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.341217 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.341636 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.341638 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.341640 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.341675 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.341916 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.341920 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.341921 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.342058 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.342154 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.342415 (+ 261us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.342519 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":38}
I20260429 08:00:48.345645 19820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Bootstrap starting.
I20260429 08:00:48.348001 19820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:48.348699 19820 log.cc:826] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:48.350449 19820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: No bootstrap required, opened a new log
I20260429 08:00:48.352797 19820 raft_consensus.cc:359] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } }
I20260429 08:00:48.353013 19820 raft_consensus.cc:385] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:48.353087 19820 raft_consensus.cc:740] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059cb565568c418e80d1a4b2d88a9477, State: Initialized, Role: FOLLOWER
I20260429 08:00:48.353468 19820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } }
I20260429 08:00:48.353891 19826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } } }
I20260429 08:00:48.354019 19826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:48.354503 19820 sys_catalog.cc:565] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: configured and running, proceeding with master startup.
W20260429 08:00:48.359472 19837 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 08:00:48.359645 19837 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 08:00:48.362758 19674 master_runner.cc:428] Detected that this master 059cb565568c418e80d1a4b2d88a9477 is joining an existing cluster
I20260429 08:00:48.362840 19674 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.12.188:40281
I20260429 08:00:48.377912 19285 heartbeater.cc:499] Master 127.12.12.190:34475 was elected leader, sending a full tablet report...
I20260429 08:00:48.383657 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.381152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35242)
0429 08:00:48.381277 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.381303 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.381387 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.381674 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.381677 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.381694 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:48.381908 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.381913 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.382616 (+ 703us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.382621 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.383208 (+ 587us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.383214 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.383342 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.383399 (+ 57us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.383458 (+ 59us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.383522 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260429 08:00:48.384094 19846 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.381198 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.12.12.190:34475)
0429 08:00:48.381497 (+ 299us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.381500 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.381509 (+ 9us) server_negotiation.cc:408] Connection header received
0429 08:00:48.381553 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.381555 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.381585 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.381667 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:48.382005 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.382507 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.383333 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.383857 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.383926 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.383964 (+ 38us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.384005 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":1408,"thread_start_us":118,"threads_started":1}
I20260429 08:00:48.389359 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.385333 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49160)
0429 08:00:48.385437 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.385452 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.385522 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.386863 (+ 1341us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.386867 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.386884 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:48.387067 (+ 183us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.387073 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.388111 (+ 1038us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.388116 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.389014 (+ 898us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.389025 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.389093 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.389107 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.389144 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.389197 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260429 08:00:48.390806 19847 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.385507 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.12.12.189:45059)
0429 08:00:48.386629 (+ 1122us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.386634 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.386645 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:48.386678 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.386681 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.386733 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.387178 (+ 445us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:48.387272 (+ 94us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.387984 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.389838 (+ 1854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.390389 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.390550 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.390613 (+ 63us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.390670 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":71,"threads_started":1}
I20260429 08:00:48.395049 19844 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.365639 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:49158)
0429 08:00:48.365897 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.365910 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.366013 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.366331 (+ 318us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.366335 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.385116 (+ 18781us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.385442 (+ 326us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.385451 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.386607 (+ 1156us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.386611 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.387178 (+ 567us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.387186 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.387275 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.388188 (+ 913us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.388203 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.388929 (+ 726us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.391602 (+ 2673us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.391605 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.391607 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.391883 (+ 276us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.392303 (+ 420us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.392306 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.392308 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.392388 (+ 80us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.392864 (+ 476us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.392867 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.393016 (+ 149us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.393185 (+ 169us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.393364 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260429 08:00:48.396267 19824 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.365438 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.12.12.189:45059)
0429 08:00:48.365578 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.365583 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.365964 (+ 381us) server_negotiation.cc:408] Connection header received
0429 08:00:48.366111 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.366114 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.366159 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.366209 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.385648 (+ 19439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.386458 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.387695 (+ 1237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.387879 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.389062 (+ 1183us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.389084 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.389087 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.389113 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.391426 (+ 2313us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.392043 (+ 617us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.392046 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.392048 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.392121 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.392507 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.392510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.392512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.392745 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.392870 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.395777 (+ 2907us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.396014 (+ 237us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 08:00:48.397545 19845 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.366051 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:35370)
0429 08:00:48.366405 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.366419 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.366500 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.366765 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.366767 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.386119 (+ 19352us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.386384 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.386392 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.387650 (+ 1258us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.387653 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.388162 (+ 509us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.388169 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.388264 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.388752 (+ 488us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.388765 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.393478 (+ 4713us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.396122 (+ 2644us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.396125 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.396128 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.396322 (+ 194us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.396569 (+ 247us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.396571 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.396573 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.396621 (+ 48us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.397000 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.397004 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.397094 (+ 90us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.397231 (+ 137us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.397343 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":72,"threads_started":1}
I20260429 08:00:48.397884 19823 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.366290 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.12.12.188:40281)
0429 08:00:48.366400 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.366404 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.366469 (+ 65us) server_negotiation.cc:408] Connection header received
0429 08:00:48.366589 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.366592 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.366634 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.366736 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.386859 (+ 20123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.387525 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.388462 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.388637 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.393604 (+ 4967us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.393623 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.393626 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.393649 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.395998 (+ 2349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.396431 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.396434 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.396436 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.396471 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.396697 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.396700 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.396702 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.396878 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.396966 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.397319 (+ 353us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.397434 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:48.404196 19843 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.365101 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35230)
0429 08:00:48.365643 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.365654 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.365761 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.366135 (+ 374us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.366138 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.385612 (+ 19474us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.385868 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.385874 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.388884 (+ 3010us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.388888 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.389430 (+ 542us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.389437 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.389851 (+ 414us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.390224 (+ 373us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.390240 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.400467 (+ 10227us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.402839 (+ 2372us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.402842 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.402844 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.402995 (+ 151us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.403291 (+ 296us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.403294 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.403296 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.403346 (+ 50us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.403704 (+ 358us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.403709 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.403794 (+ 85us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.403917 (+ 123us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.404042 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":129,"threads_started":1}
I20260429 08:00:48.404239 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.365158 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.12.12.190:34475)
0429 08:00:48.365348 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.365352 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.365722 (+ 370us) server_negotiation.cc:408] Connection header received
0429 08:00:48.365878 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.365882 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.365943 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.365997 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.385997 (+ 20000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.386749 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.389558 (+ 2809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.389739 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.400600 (+ 10861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.400616 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.400618 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.400645 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.402736 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.403083 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.403087 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.403089 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.403145 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.403437 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.403440 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.403442 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.403605 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.403698 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.403987 (+ 289us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.404091 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449647, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.190@KRBTEST.COM for oryx/127.12.12.188@KRBTEST.COM
I20260429 08:00:48.415062 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.407879 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:35374)
0429 08:00:48.408142 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.408155 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.408254 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.408516 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.408519 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.408684 (+ 165us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.408910 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.408915 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.409519 (+ 604us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.409523 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.409853 (+ 330us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.409859 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.409971 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.410437 (+ 466us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.410451 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.411867 (+ 1416us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.413953 (+ 2086us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.413956 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.413958 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.414121 (+ 163us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.414357 (+ 236us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.414360 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.414361 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.414406 (+ 45us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.414722 (+ 316us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.414724 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.414786 (+ 62us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.414862 (+ 76us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.414951 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260429 08:00:48.415282 19823 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.408003 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.12.12.188:40281)
0429 08:00:48.408118 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.408122 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.408213 (+ 91us) server_negotiation.cc:408] Connection header received
0429 08:00:48.408382 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.408384 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.408425 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.408501 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.409013 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.409420 (+ 407us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.409956 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.410152 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.411972 (+ 1820us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.411989 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.411992 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.412017 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.413851 (+ 1834us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.414216 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.414220 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.414222 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.414267 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.414487 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.414490 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.414492 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.414642 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.414732 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.414926 (+ 194us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.415025 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 08:00:48.416091 19092 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.12.12.188:40281: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "059cb565568c418e80d1a4b2d88a9477"
member_type: NON_VOTER
last_known_addr {
host: "127.12.12.188"
port: 40281
}
attrs {
promote: true
}
}
dest_uuid: "41490c9b9bf040a1acee45269d08c82d"
cas_config_opid_index: 5
I20260429 08:00:48.416585 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } }
I20260429 08:00:48.417626 19660 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.418072 19794 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:48.419027 19660 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.12.12.188:40281
I20260429 08:00:48.419196 19660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.419310 19660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:48.419452 19794 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Refusing update from remote peer 41490c9b9bf040a1acee45269d08c82d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260429 08:00:48.419751 19712 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 LEARNER]: Refusing update from remote peer 41490c9b9bf040a1acee45269d08c82d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260429 08:00:48.420228 19674 master.cc:562] Master@127.12.12.188:40281 shutting down...
I20260429 08:00:48.420228 19659 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:48.420358 19660 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260429 08:00:48.422076 19078 consensus_peers.cc:597] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d -> Peer 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188:40281): Couldn't send request to peer 059cb565568c418e80d1a4b2d88a9477. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260429 08:00:48.422475 19712 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.424075 19849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.424232 19849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: LEARNER
I20260429 08:00:48.424495 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41490c9b9bf040a1acee45269d08c82d. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: NON_VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: true } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.424588 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: LEARNER
I20260429 08:00:48.425946 19660 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: attempting to promote NON_VOTER 177399518fce4172b5943114d5d36fb5 to VOTER
I20260429 08:00:48.426357 19660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } }
I20260429 08:00:48.427505 19712 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 LEARNER]: Refusing update from remote peer 41490c9b9bf040a1acee45269d08c82d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260429 08:00:48.427887 19659 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260429 08:00:48.428339 19078 consensus_peers.cc:597] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d -> Peer 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188:40281): Couldn't send request to peer 059cb565568c418e80d1a4b2d88a9477. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20260429 08:00:48.428352 19092 master.cc:680] Remote error: unable to get registration information for peer 059cb565568c418e80d1a4b2d88a9477: Service unavailable: service kudu.master.MasterService not registered on Master
I20260429 08:00:48.430778 19659 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 177399518fce4172b5943114d5d36fb5 (127.12.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.431232 19712 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 177399518fce4172b5943114d5d36fb5 (127.12.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.432230 19659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.432318 19659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:48.432451 19659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.432523 19659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:48.432722 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.432880 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:48.434378 19674 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:48.434628 19674 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:48.434721 19674 tablet_replica.cc:333] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: stopping tablet replica
I20260429 08:00:48.449155 19674 master.cc:584] Master@127.12.12.188:40281 shutdown complete.
I20260429 08:00:48.449220 19674 master_runner.cc:305] Clearing existing system tablet
I20260429 08:00:48.449457 19674 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:48.449456 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.449170 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:35384)
0429 08:00:48.449315 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.449366 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.188:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260429 08:00:48.449514 19674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260429 08:00:48.449692 19092 master.cc:680] Network error: unable to get registration information for peer 059cb565568c418e80d1a4b2d88a9477: Client connection negotiation failed: client connection to 127.12.12.188:40281: connect: Connection refused (error 111)
I20260429 08:00:48.450765 19674 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260429 08:00:48.451637 19853 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:48.451882 19674 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:48.452023 19674 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.452073 19674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
uuid: "059cb565568c418e80d1a4b2d88a9477"
format_stamp: "Formatted at 2026-04-29 08:00:48 on dist-test-slave-nhqz"
server_key: "b23b8ff0e159f28ddc113c01ed47aee8"
server_key_iv: "42db3db858a2403f175cd33169b913ed"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.453819 19674 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:48.455655 19674 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260429 08:00:48.455729 19674 log.cc:1199] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260429 08:00:48.455992 19674 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Deleting consensus metadata
I20260429 08:00:48.456161 19674 master_runner.cc:315] Copying system tablet from 127.12.12.190:34475
I20260429 08:00:48.457314 19674 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: tablet copy: Beginning tablet copy session from remote peer at address 127.12.12.190:34475
I20260429 08:00:48.464263 19131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.457916 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.12.12.190:34475)
0429 08:00:48.458047 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.458056 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.458157 (+ 101us) server_negotiation.cc:408] Connection header received
0429 08:00:48.458281 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.458284 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.458328 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.458393 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:48.459045 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.459522 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.460360 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.460508 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.461450 (+ 942us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:48.461464 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:48.461466 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:48.461487 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:48.462790 (+ 1303us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.463230 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.463268 (+ 38us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.463270 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.463322 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:48.463560 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:48.463562 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:48.463564 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:48.463741 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:48.463834 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.464065 (+ 231us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.464147 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 08:00:48.464330 19859 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.457809 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35254)
0429 08:00:48.458070 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.458085 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.458196 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.458433 (+ 237us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.458437 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.458646 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:48.458940 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.458947 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.459631 (+ 684us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.459635 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.460234 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.460242 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.460365 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.460940 (+ 575us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:48.460956 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:48.461340 (+ 384us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:48.462890 (+ 1550us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.462894 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.462897 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.463137 (+ 240us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.463406 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:48.463409 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:48.463413 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:48.463471 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:48.463830 (+ 359us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:48.463834 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:48.463916 (+ 82us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.464023 (+ 107us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.464144 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260429 08:00:48.464772 19112 tablet_copy_service.cc:140] P 41490c9b9bf040a1acee45269d08c82d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 059cb565568c418e80d1a4b2d88a9477 ({username='oryx', principal='oryx/127.12.12.188@KRBTEST.COM'} at 127.0.0.1:35254)
I20260429 08:00:48.464893 19112 tablet_copy_service.cc:161] P 41490c9b9bf040a1acee45269d08c82d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 059cb565568c418e80d1a4b2d88a9477 at {username='oryx', principal='oryx/127.12.12.188@KRBTEST.COM'} at 127.0.0.1:35254: session id = 059cb565568c418e80d1a4b2d88a9477-00000000000000000000000000000000
I20260429 08:00:48.465896 19112 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Tablet Copy: opened 0 blocks and 1 log segments
I20260429 08:00:48.467064 19674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260429 08:00:48.468737 19674 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: tablet copy: Starting download of 0 data blocks...
I20260429 08:00:48.468948 19674 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: tablet copy: Starting download of 1 WAL segments...
I20260429 08:00:48.470427 19674 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260429 08:00:48.471814 19112 tablet_copy_service.cc:342] P 41490c9b9bf040a1acee45269d08c82d: Request end of tablet copy session 059cb565568c418e80d1a4b2d88a9477-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.12.188@KRBTEST.COM'} at 127.0.0.1:35254
I20260429 08:00:48.471901 19112 tablet_copy_service.cc:434] P 41490c9b9bf040a1acee45269d08c82d: ending tablet copy session 059cb565568c418e80d1a4b2d88a9477-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 059cb565568c418e80d1a4b2d88a9477
W20260429 08:00:48.476280 19674 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260429 08:00:48.478333 19674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:48.479992 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:48.480085 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:48.480190 19674 server_base.cc:1061] running on GCE node
W20260429 08:00:48.480962 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:48.481177 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:48.481463 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:48.482612 19674 hybrid_clock.cc:648] HybridClock initialized: now 1777449648482602 us; error 31 us; skew 500 ppm
Apr 29 08:00:48 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449648, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:48.484462 19674 init.cc:377] Logged in from keytab as oryx/127.12.12.188@KRBTEST.COM (short username oryx)
I20260429 08:00:48.484947 19674 webserver.cc:492] Webserver started at http://127.12.12.188:43687/ using document root <none> and password file <none>
I20260429 08:00:48.485133 19674 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:48.485183 19674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:48.485936 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.485700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:35386)
0429 08:00:48.485822 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.485867 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.188:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260429 08:00:48.486119 19092 master.cc:680] Network error: unable to get registration information for peer 059cb565568c418e80d1a4b2d88a9477: Client connection negotiation failed: client connection to 127.12.12.188:40281: connect: Connection refused (error 111)
I20260429 08:00:48.486598 19674 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.487512 19870 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:48.487748 19674 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:48.487811 19674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
uuid: "059cb565568c418e80d1a4b2d88a9477"
format_stamp: "Formatted at 2026-04-29 08:00:48 on dist-test-slave-nhqz"
server_key: "b23b8ff0e159f28ddc113c01ed47aee8"
server_key_iv: "42db3db858a2403f175cd33169b913ed"
server_key_version: "encryptionkey@0"
I20260429 08:00:48.487893 19674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:48.502883 19674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:48.504793 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:48.509537 19674 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.188:40281
I20260429 08:00:48.509560 19922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.188:40281 every 8 connection(s)
I20260429 08:00:48.510051 19674 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/master-2/data/info.pb
I20260429 08:00:48.511713 19923 sys_catalog.cc:263] Verifying existing consensus state
I20260429 08:00:48.512652 19923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Bootstrap starting.
I20260429 08:00:48.524055 19923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 08:00:48.524369 19923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Bootstrap complete.
I20260429 08:00:48.525043 19923 raft_consensus.cc:359] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } }
I20260429 08:00:48.525180 19923 raft_consensus.cc:740] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 059cb565568c418e80d1a4b2d88a9477, State: Initialized, Role: LEARNER
I20260429 08:00:48.525312 19923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } }
I20260429 08:00:48.525803 19923 sys_catalog.cc:565] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:48.525847 19926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.525991 19926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: This master's current role is: LEARNER
I20260429 08:00:48.528712 19937 catalog_manager.cc:1269] Loaded cluster ID: 8aab4ba730d34e47b2f51a81e40442ca
I20260429 08:00:48.528775 19937 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: loading cluster ID for follower catalog manager: success
I20260429 08:00:48.530616 19937 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: acquiring CA information for follower catalog manager: success
I20260429 08:00:48.531608 19937 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 08:00:48.556667 19657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.553947 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:35390)
0429 08:00:48.554105 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:48.554121 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:48.554210 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:48.554570 (+ 360us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:48.554574 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:48.554594 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:48.554772 (+ 178us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.554776 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.555668 (+ 892us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.555673 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:48.556268 (+ 595us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:48.556273 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.556396 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.556413 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 08:00:48.556463 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 08:00:48.556523 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260429 08:00:48.557180 19938 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:48.554054 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.12.12.188:40281)
0429 08:00:48.554346 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:48.554350 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:48.554361 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:48.554401 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:48.554403 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:48.554447 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:48.554553 (+ 106us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:48.554879 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.555561 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:48.556383 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:48.556867 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:48.556992 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:48.557043 (+ 51us) server_negotiation.cc:300] Negotiation successful
0429 08:00:48.557085 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260429 08:00:48.580384 19421 heartbeater.cc:499] Master 127.12.12.190:34475 was elected leader, sending a full tablet report...
I20260429 08:00:48.780081 19557 heartbeater.cc:499] Master 127.12.12.190:34475 was elected leader, sending a full tablet report...
I20260429 08:00:48.979276 19897 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260429 08:00:48.979717 19926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41490c9b9bf040a1acee45269d08c82d. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:48.979859 19926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: This master's current role is: LEARNER
I20260429 08:00:49.392654 19851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: NON_VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: true } } }
I20260429 08:00:49.392865 19851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:49.393784 19940 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: attempting to promote NON_VOTER 059cb565568c418e80d1a4b2d88a9477 to VOTER
I20260429 08:00:49.394204 19940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:49.395568 19712 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Refusing update from remote peer 41490c9b9bf040a1acee45269d08c82d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260429 08:00:49.395574 19897 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 LEARNER]: Refusing update from remote peer 41490c9b9bf040a1acee45269d08c82d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260429 08:00:49.396008 19940 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260429 08:00:49.396271 19851 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260429 08:00:49.399046 19940 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.399839 19712 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.400739 19940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.400893 19940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:49.400840 19939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.400947 19939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:49.401170 19939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.401297 19939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:49.401690 19943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.401804 19943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:49.401721 19897 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.403087 19926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41490c9b9bf040a1acee45269d08c82d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:49.403225 19926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:49.474339 19945 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.467810 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.12.12.189:45059)
0429 08:00:49.468034 (+ 224us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.468037 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.468050 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:00:49.468164 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.468166 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.468205 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.468281 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.469117 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.469626 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.470265 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.470410 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.471504 (+ 1094us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.471519 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.471522 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.471539 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.473062 (+ 1523us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.473403 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.473406 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.473407 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.473443 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.473626 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.473628 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.473629 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.473750 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.473843 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.474030 (+ 187us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.474115 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260429 08:00:49.477452 19946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.475180 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:34678)
0429 08:00:49.475435 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.475461 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.475525 (+ 64us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.475730 (+ 205us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.475733 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.475746 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:49.475898 (+ 152us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.475902 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.476565 (+ 663us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.476567 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.477170 (+ 603us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.477177 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.477270 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.477282 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.477325 (+ 43us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.477360 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
I20260429 08:00:49.477874 19947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.475279 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.12.12.190:34475)
0429 08:00:49.475529 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.475533 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.475545 (+ 12us) server_negotiation.cc:408] Connection header received
0429 08:00:49.475596 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.475599 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.475641 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.475708 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:49.475985 (+ 277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.476478 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.477276 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.477647 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.477686 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.477733 (+ 47us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.477777 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260429 08:00:49.481104 19946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.479064 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:35006)
0429 08:00:49.479180 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.479197 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.479267 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.479469 (+ 202us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.479471 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.479486 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:49.479637 (+ 151us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.479641 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.480241 (+ 600us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.480243 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.480788 (+ 545us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.480794 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.480934 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.480945 (+ 11us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.480980 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.481017 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260429 08:00:49.481513 19945 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.479102 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.12.12.189:45059)
0429 08:00:49.479215 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.479218 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.479229 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:49.479354 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.479356 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.479391 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.479449 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:49.479722 (+ 273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.480157 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.480948 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.481335 (+ 387us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.481371 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.481407 (+ 36us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.481442 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 08:00:49.484877 19946 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.482501 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:60774)
0429 08:00:49.482613 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.482626 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.482725 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.483097 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.483100 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.483123 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:49.483244 (+ 121us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.483248 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.484042 (+ 794us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.484045 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.484568 (+ 523us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.484574 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.484660 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.484670 (+ 10us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.484705 (+ 35us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.484738 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260429 08:00:49.485373 19948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.482627 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.12.12.188:40281)
0429 08:00:49.482910 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.482914 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.482925 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:49.482966 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.482969 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.483008 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.483080 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:49.483381 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.483955 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.484673 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.485152 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.485181 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.485223 (+ 42us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.485268 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":133,"threads_started":1}
I20260429 08:00:49.496673 19948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.487061 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.12.12.188:40281)
0429 08:00:49.487166 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.487169 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.487180 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:49.487270 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.487272 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.487323 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.487379 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.488091 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.488562 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.489322 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.489464 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.490767 (+ 1303us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.490788 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.490793 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.490825 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.494680 (+ 3855us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.495203 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.495209 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.495213 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.495441 (+ 228us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.495760 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.495766 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.495769 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.495965 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.496109 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.496334 (+ 225us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.496480 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 08:00:49.503369 19949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.499382 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:34684)
0429 08:00:49.499777 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.499794 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.499897 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.500279 (+ 382us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.500282 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.500301 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:49.500570 (+ 269us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.500580 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.501979 (+ 1399us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.501983 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.502957 (+ 974us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.502966 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.503087 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.503102 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.503167 (+ 65us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.503228 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":221,"threads_started":1}
I20260429 08:00:49.503858 19947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.499858 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.12.12.190:34475)
0429 08:00:49.499996 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.500000 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.500015 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:00:49.500087 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.500090 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.500137 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.500252 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:49.500696 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.501771 (+ 1075us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.503073 (+ 1302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.503568 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.503607 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.503659 (+ 52us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.503711 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260429 08:00:49.508656 19949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.505203 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.0.0.1:35012)
0429 08:00:49.505377 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.505393 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.505476 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.505830 (+ 354us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.505835 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.505863 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:49.506070 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.506081 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.507239 (+ 1158us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.507243 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.508278 (+ 1035us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.508289 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.508408 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.508427 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.508477 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.508538 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260429 08:00:49.509457 19945 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.505289 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.12.12.189:45059)
0429 08:00:49.505464 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.505471 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.505488 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:00:49.505625 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.505631 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.505694 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.505783 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:49.506189 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.507105 (+ 916us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.508409 (+ 1304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.509134 (+ 725us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.509181 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.509242 (+ 61us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.509309 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:00:49.513727 19949 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.510678 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.0.0.1:60802)
0429 08:00:49.510822 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.510834 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.510900 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.511201 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.511206 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.511223 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:49.511434 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.511443 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.512524 (+ 1081us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.512528 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.513355 (+ 827us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.513364 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.513492 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.513508 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.513561 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.513610 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260429 08:00:49.514078 19948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.510773 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.12.12.188:40281)
0429 08:00:49.510889 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.510895 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.510912 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:00:49.510966 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.510971 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.511031 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.511114 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:49.511566 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.512381 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.513483 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.513908 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.513934 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.513972 (+ 38us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.514009 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 08:00:49.515378 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19156
I20260429 08:00:49.522636 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:36893
--local_ip_for_outbound_sockets=127.12.12.129
--tserver_master_addrs=127.12.12.190:34475
--webserver_port=37317
--webserver_interface=127.12.12.129
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:49.635802 19950 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:49.636018 19950 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:49.636096 19950 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:49.639562 19950 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:49.639622 19950 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:49.639748 19950 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:49.644039 19950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:36893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=37317
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:49.645184 19950 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:49.646020 19950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:49.652428 19956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:49.652468 19955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:49.652627 19950 server_base.cc:1061] running on GCE node
W20260429 08:00:49.652452 19958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:49.653136 19950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:49.653676 19950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:49.654850 19950 hybrid_clock.cc:648] HybridClock initialized: now 1777449649654826 us; error 42 us; skew 500 ppm
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:49.657665 19950 init.cc:377] Logged in from keytab as oryx/127.12.12.129@KRBTEST.COM (short username oryx)
I20260429 08:00:49.658680 19950 webserver.cc:492] Webserver started at http://127.12.12.129:37317/ using document root <none> and password file <none>
I20260429 08:00:49.659198 19950 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:49.659281 19950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:49.662501 19950 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:00:49.664175 19965 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:49.665149 19950 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:49.665266 19950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "fad2a529f4a346358315acf2866329d1"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "194768745bef44922c4118808eadc729"
server_key_iv: "01cc7a0710ba581dcb57268d4acea97d"
server_key_version: "encryptionkey@0"
I20260429 08:00:49.665663 19950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:49.674985 19950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:49.677383 19950 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:49.677587 19950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:49.678097 19950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:49.678998 19950 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:49.679069 19950 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:49.679136 19950 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:49.679180 19950 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:49.687883 19950 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:36893
I20260429 08:00:49.687902 20078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:36893 every 8 connection(s)
I20260429 08:00:49.688984 19950 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.188@KRBTEST.COM
I20260429 08:00:49.698751 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 19950
I20260429 08:00:49.698935 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19292
I20260429 08:00:49.702700 19947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.690697 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:55911 (local address 127.12.12.190:34475)
0429 08:00:49.690834 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.690837 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.691523 (+ 686us) server_negotiation.cc:408] Connection header received
0429 08:00:49.692452 (+ 929us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.692455 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.692506 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.692556 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.694248 (+ 1692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695005 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.695780 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695996 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.698115 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.698133 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.698135 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.698160 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.700497 (+ 2337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.701108 (+ 611us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.701112 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.701114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.701165 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.701458 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.701462 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.701464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.701604 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.701908 (+ 304us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.702355 (+ 447us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.702526 (+ 171us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 08:00:49.703290 20083 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.690961 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:55911)
0429 08:00:49.691387 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.691417 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.692234 (+ 817us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.692949 (+ 715us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.692960 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.693878 (+ 918us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:49.694104 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.694113 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695131 (+ 1018us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.695135 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.695622 (+ 487us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.695630 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695784 (+ 154us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.696214 (+ 430us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:49.696231 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:49.697909 (+ 1678us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:49.700618 (+ 2709us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.700624 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.700635 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.700980 (+ 345us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.701259 (+ 279us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.701262 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.701265 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.701365 (+ 100us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.701907 (+ 542us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:49.701913 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:49.702115 (+ 202us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.702312 (+ 197us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.702556 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260429 08:00:49.704396 20080 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34475
I20260429 08:00:49.704663 20080 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:49.705266 20080 heartbeater.cc:507] Master 127.12.12.190:34475 requested a full tablet report, sending...
I20260429 08:00:49.705348 19948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.691947 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:50943 (local address 127.12.12.188:40281)
0429 08:00:49.692094 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.692100 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.692219 (+ 119us) server_negotiation.cc:408] Connection header received
0429 08:00:49.692440 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.692444 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.692498 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.692552 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.694189 (+ 1637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.694706 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.695458 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695647 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.699893 (+ 4246us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.699917 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.699924 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.699958 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.702626 (+ 2668us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704081 (+ 1455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.704084 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.704086 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.704137 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704407 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.704410 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.704412 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.704602 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.704669 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.705049 (+ 380us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.705173 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:00:49.705667 19945 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.692371 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:39347 (local address 127.12.12.189:45059)
0429 08:00:49.692496 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.692499 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.693055 (+ 556us) server_negotiation.cc:408] Connection header received
0429 08:00:49.693187 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.693194 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.693238 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.693311 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.694137 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.694893 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.695684 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695881 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.698938 (+ 3057us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.698960 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.698965 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.698997 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.703945 (+ 4948us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704450 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.704455 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.704457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.704506 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704833 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.704837 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.704839 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.705007 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.705092 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.705403 (+ 311us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.705516 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 08:00:49.705776 20084 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.691870 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.12.12.129:50943)
0429 08:00:49.692141 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.692154 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.692280 (+ 126us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.692949 (+ 669us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.692960 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.693515 (+ 555us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:49.694043 (+ 528us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.694050 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.694827 (+ 777us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.694829 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.695341 (+ 512us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.695349 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695542 (+ 193us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.696604 (+ 1062us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:49.696618 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:49.698942 (+ 2324us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:49.703749 (+ 4807us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.703753 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.703755 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.703958 (+ 203us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.704251 (+ 293us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704254 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.704256 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.704307 (+ 51us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.704753 (+ 446us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:49.704757 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:49.704857 (+ 100us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.705490 (+ 633us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.705614 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"spinlock_wait_cycles":58752,"thread_start_us":114,"threads_started":1}
I20260429 08:00:49.706413 20079 heartbeater.cc:344] Connected to a master server at 127.12.12.188:40281
I20260429 08:00:49.706499 20079 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:49.706732 20079 heartbeater.cc:507] Master 127.12.12.188:40281 requested a full tablet report, sending...
I20260429 08:00:49.706877 19092 ts_manager.cc:194] Re-registered known tserver with Master: fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893)
I20260429 08:00:49.707422 19092 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:55911
I20260429 08:00:49.707861 19887 ts_manager.cc:194] Registered new tserver with Master: fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893)
I20260429 08:00:49.707945 20085 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.692276 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.12.12.129:39347)
0429 08:00:49.692971 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.692982 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.693089 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.693367 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.693371 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.693536 (+ 165us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:49.694008 (+ 472us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.694018 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695001 (+ 983us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.695003 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.695559 (+ 556us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.695566 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.695994 (+ 428us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.696395 (+ 401us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:49.696409 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:49.698791 (+ 2382us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:49.704086 (+ 5295us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704092 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.704097 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.704328 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.704626 (+ 298us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.704629 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.704636 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.704691 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.705135 (+ 444us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:49.705141 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:49.705210 (+ 69us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.706411 (+ 1201us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.707789 (+ 1378us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"spinlock_wait_cycles":58240,"thread_start_us":80,"threads_started":1}
I20260429 08:00:49.709178 20081 heartbeater.cc:344] Connected to a master server at 127.12.12.189:45059
I20260429 08:00:49.709256 20081 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:49.709434 20081 heartbeater.cc:507] Master 127.12.12.189:45059 requested a full tablet report, sending...
I20260429 08:00:49.710130 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:39375
--local_ip_for_outbound_sockets=127.12.12.130
--tserver_master_addrs=127.12.12.190:34475
--webserver_port=39435
--webserver_interface=127.12.12.130
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 08:00:49.710480 19702 ts_manager.cc:194] Registered new tserver with Master: fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893)
W20260429 08:00:49.816138 20086 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:49.816354 20086 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:49.816445 20086 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:49.819787 20086 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:49.819850 20086 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:49.819988 20086 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:49.824266 20086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:39375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=39435
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:49.825435 20086 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:49.826282 20086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:49.832930 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:49.832926 20092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:49.832921 20091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:49.833195 20086 server_base.cc:1061] running on GCE node
I20260429 08:00:49.833652 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:49.834167 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:49.835314 20086 hybrid_clock.cc:648] HybridClock initialized: now 1777449649835299 us; error 30 us; skew 500 ppm
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:49.838022 20086 init.cc:377] Logged in from keytab as oryx/127.12.12.130@KRBTEST.COM (short username oryx)
I20260429 08:00:49.838999 20086 webserver.cc:492] Webserver started at http://127.12.12.130:39435/ using document root <none> and password file <none>
I20260429 08:00:49.839566 20086 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:49.839660 20086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:49.843048 20086 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:49.844748 20101 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:49.845794 20086 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:49.845918 20086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "f597d7c0ffe64506a90069b5bf96ce88"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "95283d67093c249fd34bdd819bdf4bf8"
server_key_iv: "f537f8c35c85a261196430665ee30688"
server_key_version: "encryptionkey@0"
I20260429 08:00:49.846323 20086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:49.887868 20086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:49.890878 20086 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:49.891091 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:49.891714 20086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:49.892638 20086 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:49.892707 20086 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:49.892776 20086 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:49.892848 20086 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260429 08:00:49.901904 20086 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:39375
I20260429 08:00:49.901964 20214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:39375 every 8 connection(s)
I20260429 08:00:49.902925 20086 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:49.906287 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 20086
I20260429 08:00:49.906478 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19428
I20260429 08:00:49.913992 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:35323
--local_ip_for_outbound_sockets=127.12.12.131
--tserver_master_addrs=127.12.12.190:34475
--webserver_port=42983
--webserver_interface=127.12.12.131
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281 with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
Apr 29 08:00:49 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449649, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.188@KRBTEST.COM
I20260429 08:00:49.920699 19945 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.906257 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41907 (local address 127.12.12.189:45059)
0429 08:00:49.906422 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.906429 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.906584 (+ 155us) server_negotiation.cc:408] Connection header received
0429 08:00:49.908657 (+ 2073us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.908664 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.908733 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.908790 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.910791 (+ 2001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.911592 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.912556 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.912745 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.916234 (+ 3489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.916258 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.916262 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.916302 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.918883 (+ 2581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919450 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.919454 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.919458 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.919508 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919775 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.919778 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.919780 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.919929 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.919993 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.920417 (+ 424us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.920536 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 08:00:49.920958 19947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.904702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:36621 (local address 127.12.12.190:34475)
0429 08:00:49.904846 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.904850 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.905715 (+ 865us) server_negotiation.cc:408] Connection header received
0429 08:00:49.908835 (+ 3120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.908839 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.908901 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.908960 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.910792 (+ 1832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.911600 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.912567 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.912748 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.916682 (+ 3934us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.916707 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.916713 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.916750 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.918905 (+ 2155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919544 (+ 639us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.919550 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.919554 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.919630 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919950 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.919957 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.919960 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.920155 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.920234 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.920626 (+ 392us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.920757 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 08:00:49.921465 20220 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.905039 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.12.12.130:42477)
0429 08:00:49.905569 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.905614 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.907760 (+ 2146us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x55abcc091290. stack: 00007f47aca6d0af 00007f47aca6d69f 00007f47ac544add 000055abb51cedce 000055abb51cee21 000055abb51d30e0 00007f47aca63f1d 00007f47aca63103 00007f47b0a9586f 00007f47b0aa7cfd 00007f47b0a9cb08 00007f47ad9a54d3 00007f47ad9a38c7 00007f47ad9e9e83 00007f47ad9eadbc 00007f47ada02640
0429 08:00:49.908432 (+ 672us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.909133 (+ 701us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.909147 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.909890 (+ 743us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:49.910617 (+ 727us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.910633 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.911858 (+ 1225us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.911865 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.912540 (+ 675us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.912551 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.912702 (+ 151us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.913738 (+ 1036us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:49.913755 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:49.916082 (+ 2327us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:49.918577 (+ 2495us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.918585 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.918599 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.918960 (+ 361us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.919232 (+ 272us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919238 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.919239 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.919391 (+ 152us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.919837 (+ 446us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:49.919842 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:49.920114 (+ 272us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.920351 (+ 237us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.920592 (+ 241us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":35,"spinlock_wait_cycles":2961792,"thread_start_us":127,"threads_started":1}
I20260429 08:00:49.921482 20221 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.906077 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.12.12.130:41907)
0429 08:00:49.906499 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.906514 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.908411 (+ 1897us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.909133 (+ 722us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.909145 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.909895 (+ 750us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:49.910616 (+ 721us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.910631 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.911788 (+ 1157us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.911791 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.912419 (+ 628us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.912427 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.912928 (+ 501us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.913609 (+ 681us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:49.913624 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:49.916076 (+ 2452us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:49.919005 (+ 2929us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919011 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.919016 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.919333 (+ 317us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.919616 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919619 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.919621 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.919668 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.920057 (+ 389us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:49.920060 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:49.920115 (+ 55us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.920927 (+ 812us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.921109 (+ 182us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"spinlock_wait_cycles":23808,"thread_start_us":82,"threads_started":1}
I20260429 08:00:49.921665 20219 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.905039 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:36621)
0429 08:00:49.905569 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:49.905614 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:49.908684 (+ 3070us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:49.909132 (+ 448us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:49.909145 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:49.910103 (+ 958us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:49.910616 (+ 513us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.910631 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.911788 (+ 1157us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.911791 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:49.912419 (+ 628us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:49.912427 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.912928 (+ 501us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.913470 (+ 542us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:49.913500 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:49.916507 (+ 3007us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:49.919052 (+ 2545us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919057 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.919061 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.919408 (+ 347us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.919770 (+ 362us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919776 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:49.919779 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:49.919843 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:49.920313 (+ 470us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:49.920319 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:49.920415 (+ 96us) client_negotiation.cc:770] Sending connection context
0429 08:00:49.921108 (+ 693us) client_negotiation.cc:241] Negotiation successful
0429 08:00:49.921244 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"spinlock_wait_cycles":2176,"thread_start_us":144,"threads_started":1}
I20260429 08:00:49.921957 19948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:49.904629 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:42477 (local address 127.12.12.188:40281)
0429 08:00:49.904780 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:49.904785 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:49.905716 (+ 931us) server_negotiation.cc:408] Connection header received
0429 08:00:49.908760 (+ 3044us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:49.908767 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:49.908865 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:49.908945 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:49.910792 (+ 1847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.911631 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:49.913195 (+ 1564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:49.913386 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:49.916252 (+ 2866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:49.916276 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:49.916282 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:49.916318 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:49.918409 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919066 (+ 657us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.919071 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.919073 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.919125 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:49.919489 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:49.919492 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:49.919494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:49.919701 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:49.919787 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:49.921739 (+ 1952us) server_negotiation.cc:300] Negotiation successful
0429 08:00:49.921854 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 08:00:49.922801 20216 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34475
I20260429 08:00:49.922801 20215 heartbeater.cc:344] Connected to a master server at 127.12.12.188:40281
I20260429 08:00:49.922874 20217 heartbeater.cc:344] Connected to a master server at 127.12.12.189:45059
I20260429 08:00:49.923121 20215 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:49.923125 20217 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:49.923125 20216 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:49.923779 20216 heartbeater.cc:507] Master 127.12.12.190:34475 requested a full tablet report, sending...
I20260429 08:00:49.923806 20217 heartbeater.cc:507] Master 127.12.12.189:45059 requested a full tablet report, sending...
I20260429 08:00:49.923791 20215 heartbeater.cc:507] Master 127.12.12.188:40281 requested a full tablet report, sending...
I20260429 08:00:49.925143 19887 ts_manager.cc:194] Registered new tserver with Master: f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375)
I20260429 08:00:49.925143 19702 ts_manager.cc:194] Registered new tserver with Master: f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375)
I20260429 08:00:49.925143 19092 ts_manager.cc:194] Re-registered known tserver with Master: f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375)
I20260429 08:00:49.925937 19092 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:36621
W20260429 08:00:50.022003 20222 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:50.022234 20222 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:50.022291 20222 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:50.025691 20222 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:50.025758 20222 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:50.025840 20222 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:50.030287 20222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:35323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=42983
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34475,127.12.12.189:45059,127.12.12.188:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:50.031320 20222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:50.032259 20222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:50.038968 20229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:50.038961 20228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:50.039330 20222 server_base.cc:1061] running on GCE node
W20260429 08:00:50.038964 20231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:50.039822 20222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:50.040375 20222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:50.041541 20222 hybrid_clock.cc:648] HybridClock initialized: now 1777449650041507 us; error 51 us; skew 500 ppm
Apr 29 08:00:50 dist-test-slave-nhqz krb5kdc[19044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449650, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:50.044194 20222 init.cc:377] Logged in from keytab as oryx/127.12.12.131@KRBTEST.COM (short username oryx)
I20260429 08:00:50.045233 20222 webserver.cc:492] Webserver started at http://127.12.12.131:42983/ using document root <none> and password file <none>
I20260429 08:00:50.045794 20222 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:50.045874 20222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:50.049353 20222 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:00:50.051198 20238 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:50.052151 20222 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:50.052292 20222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "57838a88843949328cd35e764aa23bb8"
format_stamp: "Formatted at 2026-04-29 08:00:47 on dist-test-slave-nhqz"
server_key: "9d6f7a6b4a30a11d59e94af1f98e6404"
server_key_iv: "0a0184282455a5d7024ab7e7f14bdc22"
server_key_version: "encryptionkey@0"
I20260429 08:00:50.052675 20222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:50.069914 20222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:50.072403 20222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:50.072674 20222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:50.073300 20222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:00:50.074147 20222 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:50.074213 20222 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:50.074283 20222 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:50.074342 20222 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:50.084245 20222 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:35323
I20260429 08:00:50.084290 20351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:35323 every 8 connection(s)
I20260429 08:00:50.085287 20222 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:50.090660 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 20222
Apr 29 08:00:50 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449650, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
Apr 29 08:00:50 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449650, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.189@KRBTEST.COM
Apr 29 08:00:50 dist-test-slave-nhqz krb5kdc[19044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449650, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.188@KRBTEST.COM
I20260429 08:00:50.100553 19945 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.088367 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:47759 (local address 127.12.12.189:45059)
0429 08:00:50.088497 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.088501 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.088650 (+ 149us) server_negotiation.cc:408] Connection header received
0429 08:00:50.089255 (+ 605us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.089259 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.089305 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.089383 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.090615 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091259 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.092136 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.092296 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.095335 (+ 3039us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.095356 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.095360 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.095387 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.098719 (+ 3332us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.099130 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.099136 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.099140 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.099199 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.099434 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.099440 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.099444 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.099599 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.099680 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.100302 (+ 622us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.100399 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:50.101230 20358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.088270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.189:45059 (local address 127.12.12.131:47759)
0429 08:00:50.088576 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.088587 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.088732 (+ 145us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.089413 (+ 681us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.089416 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.090217 (+ 801us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:50.090479 (+ 262us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.090488 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091493 (+ 1005us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.091498 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.092003 (+ 505us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.092012 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.092391 (+ 379us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.093067 (+ 676us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:50.093094 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:50.095159 (+ 2065us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:50.098828 (+ 3669us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:50.098831 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:50.098833 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:50.099030 (+ 197us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:50.099290 (+ 260us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:50.099293 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:50.099295 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:50.099338 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:50.099717 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:50.099720 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:50.099822 (+ 102us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.100071 (+ 249us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.100855 (+ 784us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":535,"spinlock_wait_cycles":4096,"thread_start_us":127,"threads_started":1}
I20260429 08:00:50.101227 20356 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.087297 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.188:40281 (local address 127.12.12.131:38159)
0429 08:00:50.087733 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.087770 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.088713 (+ 943us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.089267 (+ 554us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.089275 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.089827 (+ 552us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:50.090242 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.090251 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091223 (+ 972us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.091227 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.091772 (+ 545us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.091779 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091974 (+ 195us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.093332 (+ 1358us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:50.093350 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:50.095738 (+ 2388us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:50.098362 (+ 2624us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:50.098370 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:50.098383 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:50.098710 (+ 327us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:50.098983 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:50.098986 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:50.098990 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:50.099115 (+ 125us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:50.099543 (+ 428us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:50.099548 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:50.099822 (+ 274us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.100071 (+ 249us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.100323 (+ 252us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":34048,"thread_start_us":95,"threads_started":1}
I20260429 08:00:50.101958 19948 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.086988 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:38159 (local address 127.12.12.188:40281)
0429 08:00:50.087133 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.087138 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.087863 (+ 725us) server_negotiation.cc:408] Connection header received
0429 08:00:50.089013 (+ 1150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.089016 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.089067 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.089137 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.090379 (+ 1242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091086 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.092783 (+ 1697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.093024 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.095871 (+ 2847us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.095891 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.095898 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.095933 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.098216 (+ 2283us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.098830 (+ 614us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.098834 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.098835 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.098883 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.099223 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.099229 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.099233 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.099413 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.099494 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.101672 (+ 2178us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.101799 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:00:50.102656 20354 heartbeater.cc:344] Connected to a master server at 127.12.12.189:45059
I20260429 08:00:50.102916 20354 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:50.103049 20352 heartbeater.cc:344] Connected to a master server at 127.12.12.188:40281
I20260429 08:00:50.103145 20352 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:50.103559 20354 heartbeater.cc:507] Master 127.12.12.189:45059 requested a full tablet report, sending...
I20260429 08:00:50.103559 20352 heartbeater.cc:507] Master 127.12.12.188:40281 requested a full tablet report, sending...
I20260429 08:00:50.105129 20357 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.087298 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:44619)
0429 08:00:50.087733 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.087771 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.088729 (+ 958us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.089266 (+ 537us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.089274 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.089831 (+ 557us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:50.090268 (+ 437us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.090275 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091223 (+ 948us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.091227 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.091772 (+ 545us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.091779 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.092223 (+ 444us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.092661 (+ 438us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:50.092684 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:50.095530 (+ 2846us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:50.101274 (+ 5744us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:50.101280 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:50.101285 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:50.101496 (+ 211us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:50.104131 (+ 2635us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:50.104134 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:50.104136 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:50.104189 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:50.104641 (+ 452us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:50.104645 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:50.104725 (+ 80us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.104852 (+ 127us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.104969 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":59,"spinlock_wait_cycles":8192,"thread_start_us":108,"threads_started":1}
I20260429 08:00:50.105108 19887 ts_manager.cc:194] Registered new tserver with Master: 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323)
I20260429 08:00:50.105485 20364 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.092794 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.12.12.189:45059)
0429 08:00:50.094847 (+ 2053us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.094850 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.094859 (+ 9us) server_negotiation.cc:408] Connection header received
0429 08:00:50.094885 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.094888 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.094921 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.095016 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.096349 (+ 1333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.097180 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.098359 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.098553 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.099907 (+ 1354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.099925 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.099928 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.099958 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.103236 (+ 3278us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.103732 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.103736 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.103738 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.103782 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.104004 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.104007 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.104009 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.104154 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.104277 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.104897 (+ 620us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.105000 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1990,"thread_start_us":97,"threads_started":1}
I20260429 08:00:50.105685 19947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.087119 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:44619 (local address 127.12.12.190:34475)
0429 08:00:50.087248 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.087253 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.087864 (+ 611us) server_negotiation.cc:408] Connection header received
0429 08:00:50.088959 (+ 1095us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.088964 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.089025 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.089132 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.090446 (+ 1314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.091110 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.091919 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.092103 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.095662 (+ 3559us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.095687 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.095693 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.095728 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.097788 (+ 2060us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.102879 (+ 5091us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.102883 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.102885 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.102946 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.104301 (+ 1355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.104304 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.104306 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.104504 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.104596 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.105363 (+ 767us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.105505 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 08:00:50.105707 19702 ts_manager.cc:194] Registered new tserver with Master: 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323)
I20260429 08:00:50.106248 20353 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34475
I20260429 08:00:50.106318 20353 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:50.106503 20353 heartbeater.cc:507] Master 127.12.12.190:34475 requested a full tablet report, sending...
I20260429 08:00:50.107082 19092 ts_manager.cc:194] Re-registered known tserver with Master: 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323)
I20260429 08:00:50.107769 19092 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:44619
I20260429 08:00:50.113344 20367 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.093755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.12.12.190:34475)
0429 08:00:50.093993 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.093997 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.094478 (+ 481us) server_negotiation.cc:408] Connection header received
0429 08:00:50.094531 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.094534 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.094594 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.106738 (+ 12144us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.107689 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.108533 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.109400 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.109580 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.110535 (+ 955us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.110552 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.110555 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.110577 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.112032 (+ 1455us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.112382 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.112385 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.112387 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.112430 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.112608 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.112611 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.112613 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.112757 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.112895 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.113070 (+ 175us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.113175 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260429 08:00:50.117762 20365 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.093281 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.12.12.188:40281)
0429 08:00:50.094246 (+ 965us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.094251 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.094718 (+ 467us) server_negotiation.cc:408] Connection header received
0429 08:00:50.095241 (+ 523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.095245 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.095291 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.095365 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.096868 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.097641 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.098491 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.098669 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.100171 (+ 1502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.100191 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.100196 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.100222 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.102467 (+ 2245us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.102948 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.102952 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.102954 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.103002 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.103297 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.103302 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.103304 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.117057 (+ 13753us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.117207 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.117457 (+ 250us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.117602 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":77,"threads_started":1}
I20260429 08:00:50.115935 19092 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:50.119045 19092 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:50.134889 20377 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.130689 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:35323 (local address 127.0.0.1:41926)
0429 08:00:50.130974 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.130990 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.131078 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.132289 (+ 1211us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.132292 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.132312 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:50.132544 (+ 232us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.132550 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.133652 (+ 1102us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.133658 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.134514 (+ 856us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.134523 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.134633 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.134649 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.134695 (+ 46us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.134750 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":133,"threads_started":1}
I20260429 08:00:50.135485 20380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.131055 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:36893 (local address 127.0.0.1:54204)
0429 08:00:50.131996 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.132009 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.132084 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.132654 (+ 570us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.132657 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.132676 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:50.132933 (+ 257us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.132943 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.134345 (+ 1402us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.134351 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.135130 (+ 779us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.135137 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.135240 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.135253 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.135296 (+ 43us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.135354 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":80,"threads_started":1}
I20260429 08:00:50.136212 20381 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.131439 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.12.12.129:36893)
0429 08:00:50.132302 (+ 863us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.132309 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.132342 (+ 33us) server_negotiation.cc:408] Connection header received
0429 08:00:50.132413 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.132416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.132528 (+ 112us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.132668 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:50.133061 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.134086 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.135278 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.135823 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.135939 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.136000 (+ 61us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.136078 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":86,"threads_started":1}
I20260429 08:00:50.136230 20379 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.131323 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:39375 (local address 127.0.0.1:34554)
0429 08:00:50.132926 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.132938 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.133022 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.133569 (+ 547us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.133572 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.133591 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:50.133811 (+ 220us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.133816 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.135125 (+ 1309us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.135129 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.135883 (+ 754us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.135891 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.136012 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.136026 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.136073 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.136121 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1489,"thread_start_us":75,"threads_started":1}
I20260429 08:00:50.136687 20378 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.130829 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.12.12.131:35323)
0429 08:00:50.131340 (+ 511us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.131349 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.131371 (+ 22us) server_negotiation.cc:408] Connection header received
0429 08:00:50.131969 (+ 598us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.131974 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.132139 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.132296 (+ 157us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:50.132657 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.133511 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.135414 (+ 1903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.136077 (+ 663us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.136162 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.136222 (+ 60us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.136549 (+ 327us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":103,"threads_started":1}
I20260429 08:00:50.136878 20382 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.131683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.12.12.130:39375)
0429 08:00:50.132303 (+ 620us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.132309 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.133106 (+ 797us) server_negotiation.cc:408] Connection header received
0429 08:00:50.133186 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.133191 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.133322 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.133446 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:50.133962 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.134995 (+ 1033us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.136018 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.136549 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.136622 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.136671 (+ 49us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.136723 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":91,"threads_started":1}
I20260429 08:00:50.138984 20013 tablet_service.cc:1511] Processing CreateTablet for tablet 075e4382b315415d868e4676ac3f7419 (DEFAULT_TABLE table=test-table [id=d6d941c1ef5a4697b5ccf33c2c1d3be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:50.139070 20286 tablet_service.cc:1511] Processing CreateTablet for tablet 075e4382b315415d868e4676ac3f7419 (DEFAULT_TABLE table=test-table [id=d6d941c1ef5a4697b5ccf33c2c1d3be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:50.139855 20013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075e4382b315415d868e4676ac3f7419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:50.139858 20286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075e4382b315415d868e4676ac3f7419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:50.145532 20383 tablet_bootstrap.cc:492] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Bootstrap starting.
I20260429 08:00:50.146747 20149 tablet_service.cc:1511] Processing CreateTablet for tablet 075e4382b315415d868e4676ac3f7419 (DEFAULT_TABLE table=test-table [id=d6d941c1ef5a4697b5ccf33c2c1d3be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:50.147672 20149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 075e4382b315415d868e4676ac3f7419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:50.147701 20383 tablet_bootstrap.cc:654] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:50.148468 20383 log.cc:826] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:50.149515 20385 tablet_bootstrap.cc:492] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Bootstrap starting.
I20260429 08:00:50.151105 20383 tablet_bootstrap.cc:492] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: No bootstrap required, opened a new log
I20260429 08:00:50.151291 20383 ts_tablet_manager.cc:1403] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260429 08:00:50.153823 20386 tablet_bootstrap.cc:492] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Bootstrap starting.
I20260429 08:00:50.154363 20383 raft_consensus.cc:359] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.154529 20383 raft_consensus.cc:385] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:50.154614 20383 raft_consensus.cc:740] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad2a529f4a346358315acf2866329d1, State: Initialized, Role: FOLLOWER
I20260429 08:00:50.154989 20383 consensus_queue.cc:260] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.155612 20080 heartbeater.cc:499] Master 127.12.12.190:34475 was elected leader, sending a full tablet report...
I20260429 08:00:50.155736 20383 ts_tablet_manager.cc:1434] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260429 08:00:50.155903 20386 tablet_bootstrap.cc:654] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:50.156708 20386 log.cc:826] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:50.158684 20386 tablet_bootstrap.cc:492] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: No bootstrap required, opened a new log
I20260429 08:00:50.158852 20386 ts_tablet_manager.cc:1403] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:00:50.160660 20385 tablet_bootstrap.cc:654] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:50.161502 20385 log.cc:826] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:50.161522 20386 raft_consensus.cc:359] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.161756 20386 raft_consensus.cc:385] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:50.161814 20386 raft_consensus.cc:740] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f597d7c0ffe64506a90069b5bf96ce88, State: Initialized, Role: FOLLOWER
I20260429 08:00:50.162190 20386 consensus_queue.cc:260] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.162793 20216 heartbeater.cc:499] Master 127.12.12.190:34475 was elected leader, sending a full tablet report...
I20260429 08:00:50.162901 20386 ts_tablet_manager.cc:1434] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260429 08:00:50.163247 20385 tablet_bootstrap.cc:492] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: No bootstrap required, opened a new log
I20260429 08:00:50.163412 20385 ts_tablet_manager.cc:1403] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260429 08:00:50.166302 20385 raft_consensus.cc:359] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.166514 20385 raft_consensus.cc:385] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:50.166601 20385 raft_consensus.cc:740] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57838a88843949328cd35e764aa23bb8, State: Initialized, Role: FOLLOWER
I20260429 08:00:50.167049 20385 consensus_queue.cc:260] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.167644 20353 heartbeater.cc:499] Master 127.12.12.190:34475 was elected leader, sending a full tablet report...
I20260429 08:00:50.167822 20385 ts_tablet_manager.cc:1434] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260429 08:00:50.189339 20391 raft_consensus.cc:493] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:50.189533 20391 raft_consensus.cc:515] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.190694 20391 leader_election.cc:290] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893), f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375)
W20260429 08:00:50.190851 20082 tablet.cc:2404] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:50.193560 20357 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.190914 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:36893 (local address 127.12.12.131:48991)
0429 08:00:50.191031 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.191046 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.191131 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.191575 (+ 444us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.191578 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.191609 (+ 31us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:50.191823 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.191828 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.192538 (+ 710us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.192543 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.193196 (+ 653us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.193204 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.193317 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.193333 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.193381 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.193431 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260429 08:00:50.193697 20358 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.190919 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:39375 (local address 127.12.12.131:40221)
0429 08:00:50.191366 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:50.191380 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:50.191492 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:50.191775 (+ 283us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:50.191778 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:50.191794 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:50.191970 (+ 176us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.191974 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.192675 (+ 701us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.192680 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:50.193365 (+ 685us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:50.193373 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.193465 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.193481 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 08:00:50.193526 (+ 45us) client_negotiation.cc:241] Negotiation successful
0429 08:00:50.193588 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":283}
I20260429 08:00:50.194061 20381 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.191222 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:48991 (local address 127.12.12.129:36893)
0429 08:00:50.191363 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.191371 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.191384 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:00:50.191427 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.191429 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.191469 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.191556 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:50.191937 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.192426 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.193309 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.193769 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.193812 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.193865 (+ 53us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.193926 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 08:00:50.194617 20382 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.191125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:40221 (local address 127.12.12.130:39375)
0429 08:00:50.191260 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.191265 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.191451 (+ 186us) server_negotiation.cc:408] Connection header received
0429 08:00:50.191627 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.191630 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.191685 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.191765 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:50.192079 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.192563 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.193855 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.194369 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.194408 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.194457 (+ 49us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.194512 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 08:00:50.194628 20033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075e4382b315415d868e4676ac3f7419" candidate_uuid: "57838a88843949328cd35e764aa23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad2a529f4a346358315acf2866329d1" is_pre_election: true
I20260429 08:00:50.194921 20033 raft_consensus.cc:2468] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57838a88843949328cd35e764aa23bb8 in term 0.
I20260429 08:00:50.195154 20169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075e4382b315415d868e4676ac3f7419" candidate_uuid: "57838a88843949328cd35e764aa23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f597d7c0ffe64506a90069b5bf96ce88" is_pre_election: true
I20260429 08:00:50.195410 20169 raft_consensus.cc:2468] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57838a88843949328cd35e764aa23bb8 in term 0.
I20260429 08:00:50.195402 20241 leader_election.cc:304] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57838a88843949328cd35e764aa23bb8, fad2a529f4a346358315acf2866329d1; no voters:
I20260429 08:00:50.195683 20391 raft_consensus.cc:2804] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:50.195792 20391 raft_consensus.cc:493] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:50.195838 20391 raft_consensus.cc:3060] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:50.196784 20391 raft_consensus.cc:515] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.197156 20391 leader_election.cc:290] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [CANDIDATE]: Term 1 election: Requested vote from peers fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893), f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375)
I20260429 08:00:50.197455 20033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075e4382b315415d868e4676ac3f7419" candidate_uuid: "57838a88843949328cd35e764aa23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad2a529f4a346358315acf2866329d1"
I20260429 08:00:50.197492 20169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "075e4382b315415d868e4676ac3f7419" candidate_uuid: "57838a88843949328cd35e764aa23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f597d7c0ffe64506a90069b5bf96ce88"
I20260429 08:00:50.197556 20033 raft_consensus.cc:3060] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:50.197644 20169 raft_consensus.cc:3060] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:50.198508 20033 raft_consensus.cc:2468] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57838a88843949328cd35e764aa23bb8 in term 1.
I20260429 08:00:50.198618 20169 raft_consensus.cc:2468] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57838a88843949328cd35e764aa23bb8 in term 1.
I20260429 08:00:50.198805 20241 leader_election.cc:304] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57838a88843949328cd35e764aa23bb8, fad2a529f4a346358315acf2866329d1; no voters:
I20260429 08:00:50.198982 20391 raft_consensus.cc:2804] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:50.199225 20391 raft_consensus.cc:697] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 57838a88843949328cd35e764aa23bb8, State: Running, Role: LEADER
I20260429 08:00:50.199537 20391 consensus_queue.cc:237] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } }
I20260429 08:00:50.202951 19092 catalog_manager.cc:5671] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57838a88843949328cd35e764aa23bb8 (127.12.12.131). New cstate: current_term: 1 leader_uuid: "57838a88843949328cd35e764aa23bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:50.215617 20378 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.212997 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.12.12.131:35323)
0429 08:00:50.213117 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.213120 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.213132 (+ 12us) server_negotiation.cc:408] Connection header received
0429 08:00:50.213255 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.213258 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.213308 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.213385 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:50.213694 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.214143 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.214756 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.214884 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.214954 (+ 70us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:50.215291 (+ 337us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:50.215376 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.215454 (+ 78us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.215494 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 08:00:50.221252 20169 raft_consensus.cc:1275] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Refusing update from remote peer 57838a88843949328cd35e764aa23bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:50.221293 20033 raft_consensus.cc:1275] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 1 FOLLOWER]: Refusing update from remote peer 57838a88843949328cd35e764aa23bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:50.221843 20391 consensus_queue.cc:1048] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:50.221998 20392 consensus_queue.cc:1048] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:50.233111 20397 mvcc.cc:204] Tried to move back new op lower bound from 7280433767299538944 to 7280433767219679232. Current Snapshot: MvccSnapshot[applied={T|T < 7280433767299538944 or (T in {7280433767299538944})}]
I20260429 08:00:50.238510 19092 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:50.238674 19092 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:50.244112 19092 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 41490c9b9bf040a1acee45269d08c82d: Sending DeleteTablet for 3 replicas of tablet 075e4382b315415d868e4676ac3f7419
I20260429 08:00:50.245106 20149 tablet_service.cc:1558] Processing DeleteTablet for tablet 075e4382b315415d868e4676ac3f7419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:50 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:34554
I20260429 08:00:50.245568 20013 tablet_service.cc:1558] Processing DeleteTablet for tablet 075e4382b315415d868e4676ac3f7419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:50 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:54204
I20260429 08:00:50.245600 20012 tablet_service.cc:1558] Processing DeleteTablet for tablet 075e4382b315415d868e4676ac3f7419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:50 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:54204
I20260429 08:00:50.245827 20286 tablet_service.cc:1558] Processing DeleteTablet for tablet 075e4382b315415d868e4676ac3f7419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:50 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:41926
I20260429 08:00:50.246039 20405 tablet_replica.cc:333] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: stopping tablet replica
I20260429 08:00:50.246313 20405 raft_consensus.cc:2243] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:50.246627 20405 raft_consensus.cc:2272] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:50.247143 20406 tablet_replica.cc:333] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: stopping tablet replica
I20260429 08:00:50.247400 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19060
I20260429 08:00:50.247414 20404 tablet_replica.cc:333] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: stopping tablet replica
I20260429 08:00:50.248085 20406 raft_consensus.cc:2243] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:50.248394 20404 raft_consensus.cc:2243] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:50.248682 20405 ts_tablet_manager.cc:1916] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:50.249090 20406 raft_consensus.cc:2272] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:50.249205 20404 raft_consensus.cc:2272] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:50.251366 20406 ts_tablet_manager.cc:1916] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:50.251554 20405 ts_tablet_manager.cc:1929] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:50.251859 20405 log.cc:1199] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/075e4382b315415d868e4676ac3f7419
I20260429 08:00:50.252214 20405 ts_tablet_manager.cc:1950] T 075e4382b315415d868e4676ac3f7419 P fad2a529f4a346358315acf2866329d1: Deleting consensus metadata
I20260429 08:00:50.252206 20404 ts_tablet_manager.cc:1916] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:50.254917 20404 ts_tablet_manager.cc:1929] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:50.255031 20404 log.cc:1199] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/075e4382b315415d868e4676ac3f7419
I20260429 08:00:50.255321 20404 ts_tablet_manager.cc:1950] T 075e4382b315415d868e4676ac3f7419 P f597d7c0ffe64506a90069b5bf96ce88: Deleting consensus metadata
W20260429 08:00:50.256363 19968 connection.cc:570] server connection from 127.0.0.1:54204 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 08:00:50.258843 20406 ts_tablet_manager.cc:1929] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:50.258940 20406 log.cc:1199] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/075e4382b315415d868e4676ac3f7419
I20260429 08:00:50.259264 20406 ts_tablet_manager.cc:1950] T 075e4382b315415d868e4676ac3f7419 P 57838a88843949328cd35e764aa23bb8: Deleting consensus metadata
W20260429 08:00:50.260149 20241 connection.cc:463] server connection from 127.0.0.1:41926 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41926 (request call id 1) could send its response
I20260429 08:00:50.268802 20364 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.260589 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.12.12.189:45059)
0429 08:00:50.260713 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.260717 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.260850 (+ 133us) server_negotiation.cc:408] Connection header received
0429 08:00:50.261006 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.261010 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.261064 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.261108 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.262305 (+ 1197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.263138 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.263948 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.264178 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.265641 (+ 1463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.265658 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.265660 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.265683 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.267245 (+ 1562us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.267637 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.267641 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.267642 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.267692 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.267944 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.267947 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.267949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.268116 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.268243 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.268479 (+ 236us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.268605 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 08:00:50.269358 20365 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:50.260788 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.12.12.188:40281)
0429 08:00:50.260981 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:50.260985 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:50.261401 (+ 416us) server_negotiation.cc:408] Connection header received
0429 08:00:50.261502 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:50.261506 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:50.261563 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:50.261657 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:50.262868 (+ 1211us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.263520 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:50.264412 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:50.264654 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:50.265874 (+ 1220us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:50.265893 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:50.265896 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:50.265924 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:50.267615 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.268108 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.268112 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.268114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.268160 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:50.268396 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:50.268401 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:50.268405 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:50.268589 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:50.268741 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:50.269128 (+ 387us) server_negotiation.cc:300] Negotiation successful
0429 08:00:50.269241 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260429 08:00:51.234110 20413 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.233596 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:37449)
0429 08:00:51.233912 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.233993 (+ 81us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
W20260429 08:00:51.234495 20216 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:34475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.235041 20414 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.234644 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:45329)
0429 08:00:51.234885 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.234954 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
W20260429 08:00:51.235297 20353 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:34475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.247586 20415 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.247181 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:33567)
0429 08:00:51.247416 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.247480 (+ 64us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
W20260429 08:00:51.247877 20080 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:34475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.744004 20416 raft_consensus.cc:493] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41490c9b9bf040a1acee45269d08c82d)
I20260429 08:00:51.744029 20417 raft_consensus.cc:493] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41490c9b9bf040a1acee45269d08c82d)
I20260429 08:00:51.744199 20416 raft_consensus.cc:515] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:51.744199 20417 raft_consensus.cc:515] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:51.745378 20417 leader_election.cc:290] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475), 177399518fce4172b5943114d5d36fb5 (127.12.12.189:45059)
I20260429 08:00:51.745543 20416 leader_election.cc:290] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475), 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188:40281)
I20260429 08:00:51.745796 19897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "177399518fce4172b5943114d5d36fb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "059cb565568c418e80d1a4b2d88a9477" is_pre_election: true
I20260429 08:00:51.745980 19897 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 177399518fce4172b5943114d5d36fb5 in term 1.
I20260429 08:00:51.746031 19712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "059cb565568c418e80d1a4b2d88a9477" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "177399518fce4172b5943114d5d36fb5" is_pre_election: true
I20260429 08:00:51.746183 20419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.745773 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35120)
0429 08:00:51.746003 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.746068 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":59,"threads_started":1}
I20260429 08:00:51.746191 20418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.745641 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35110)
0429 08:00:51.746041 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.746105 (+ 64us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":59,"threads_started":1}
I20260429 08:00:51.746197 19712 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059cb565568c418e80d1a4b2d88a9477 in term 1.
W20260429 08:00:51.746325 19689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.746373 19688 leader_election.cc:304] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059cb565568c418e80d1a4b2d88a9477, 177399518fce4172b5943114d5d36fb5; no voters:
W20260429 08:00:51.746445 19874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.746587 20416 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260429 08:00:51.746659 20416 raft_consensus.cc:493] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 41490c9b9bf040a1acee45269d08c82d)
I20260429 08:00:51.746707 20416 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 1 FOLLOWER]: Advancing to term 2
I20260429 08:00:51.746788 19873 leader_election.cc:304] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059cb565568c418e80d1a4b2d88a9477, 177399518fce4172b5943114d5d36fb5; no voters:
I20260429 08:00:51.747049 20417 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260429 08:00:51.747133 20417 raft_consensus.cc:493] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 41490c9b9bf040a1acee45269d08c82d)
I20260429 08:00:51.747229 20417 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 1 FOLLOWER]: Advancing to term 2
I20260429 08:00:51.747332 20419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.747035 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35148)
0429 08:00:51.747215 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.747249 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260429 08:00:51.747318 20418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.747035 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35136)
0429 08:00:51.747186 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.747228 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260429 08:00:51.747470 19874 leader_election.cc:336] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
W20260429 08:00:51.747462 19689 leader_election.cc:336] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.748050 20416 raft_consensus.cc:515] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:51.748459 20416 leader_election.cc:290] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [CANDIDATE]: Term 2 election: Requested vote from peers 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475), 059cb565568c418e80d1a4b2d88a9477 (127.12.12.188:40281)
I20260429 08:00:51.748517 20417 raft_consensus.cc:515] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:51.748764 20418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.748526 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35164)
0429 08:00:51.748658 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.748704 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260429 08:00:51.748785 19897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "177399518fce4172b5943114d5d36fb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "059cb565568c418e80d1a4b2d88a9477"
I20260429 08:00:51.748960 19897 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 177399518fce4172b5943114d5d36fb5 in current term 2: Already voted for candidate 059cb565568c418e80d1a4b2d88a9477 in this term.
I20260429 08:00:51.749315 20419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.749099 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35168)
0429 08:00:51.749212 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.749254 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260429 08:00:51.749358 20417 leader_election.cc:290] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 2 election: Requested vote from peers 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475), 177399518fce4172b5943114d5d36fb5 (127.12.12.189:45059)
I20260429 08:00:51.749498 19712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "059cb565568c418e80d1a4b2d88a9477" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "177399518fce4172b5943114d5d36fb5"
I20260429 08:00:51.749650 19712 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 059cb565568c418e80d1a4b2d88a9477 in current term 2: Already voted for candidate 177399518fce4172b5943114d5d36fb5 in this term.
I20260429 08:00:51.749985 20419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.749771 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35182)
0429 08:00:51.749891 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:51.749931 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260429 08:00:51.750100 19874 leader_election.cc:336] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:51.750193 19874 leader_election.cc:304] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059cb565568c418e80d1a4b2d88a9477; no voters: 177399518fce4172b5943114d5d36fb5, 41490c9b9bf040a1acee45269d08c82d
I20260429 08:00:51.750324 20417 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260429 08:00:52.235755 20422 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:52.235203 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:60533)
0429 08:00:52.235562 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:52.235626 (+ 64us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":170,"threads_started":1}
I20260429 08:00:52.236282 20423 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:52.235885 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:58959)
0429 08:00:52.236138 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:52.236192 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260429 08:00:52.248960 20424 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:52.248512 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:56597)
0429 08:00:52.248775 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:52.248873 (+ 98us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":127,"threads_started":1}
I20260429 08:00:52.757191 20418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:51.749225 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35174)
0429 08:00:51.749376 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:52.757081 (+1007705us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260429 08:00:52.757386 19689 leader_election.cc:336] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:52.757507 19689 leader_election.cc:304] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 177399518fce4172b5943114d5d36fb5; no voters: 059cb565568c418e80d1a4b2d88a9477, 41490c9b9bf040a1acee45269d08c82d
I20260429 08:00:52.757786 20425 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260429 08:00:53.236801 20426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.236358 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:39103)
0429 08:00:53.236623 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.236679 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260429 08:00:53.237418 20427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.236980 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:53059)
0429 08:00:53.237257 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.237312 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260429 08:00:53.249966 20428 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.249608 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:57203)
0429 08:00:53.249845 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.249889 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
I20260429 08:00:53.497905 20429 raft_consensus.cc:493] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:53.498050 20429 raft_consensus.cc:515] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:53.498602 20429 leader_election.cc:290] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475), 177399518fce4172b5943114d5d36fb5 (127.12.12.189:45059)
I20260429 08:00:53.499132 19712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "059cb565568c418e80d1a4b2d88a9477" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "177399518fce4172b5943114d5d36fb5" is_pre_election: true
I20260429 08:00:53.499298 19712 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059cb565568c418e80d1a4b2d88a9477 in term 2.
I20260429 08:00:53.499362 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.498884 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35450)
0429 08:00:53.499149 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.499200 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260429 08:00:53.499675 19873 leader_election.cc:304] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059cb565568c418e80d1a4b2d88a9477, 177399518fce4172b5943114d5d36fb5; no voters:
I20260429 08:00:53.499884 20429 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260429 08:00:53.499956 20429 raft_consensus.cc:493] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:53.500012 20429 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 2 FOLLOWER]: Advancing to term 3
I20260429 08:00:53.500337 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.500085 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35462)
0429 08:00:53.500211 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.500260 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260429 08:00:53.500469 19874 leader_election.cc:336] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:53.501194 20429 raft_consensus.cc:515] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:53.501539 20429 leader_election.cc:290] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 3 election: Requested vote from peers 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475), 177399518fce4172b5943114d5d36fb5 (127.12.12.189:45059)
I20260429 08:00:53.501891 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.501649 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35466)
0429 08:00:53.501789 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.501832 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260429 08:00:53.501924 19712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "059cb565568c418e80d1a4b2d88a9477" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "177399518fce4172b5943114d5d36fb5"
I20260429 08:00:53.502080 19712 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 2 FOLLOWER]: Advancing to term 3
I20260429 08:00:53.502471 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.502283 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35468)
0429 08:00:53.502383 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.502421 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260429 08:00:53.502570 19874 leader_election.cc:336] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
I20260429 08:00:53.503067 19712 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059cb565568c418e80d1a4b2d88a9477 in term 3.
I20260429 08:00:53.503448 19873 leader_election.cc:304] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059cb565568c418e80d1a4b2d88a9477, 177399518fce4172b5943114d5d36fb5; no voters: 41490c9b9bf040a1acee45269d08c82d
I20260429 08:00:53.503652 20429 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 3 FOLLOWER]: Leader election won for term 3
I20260429 08:00:53.503834 20429 raft_consensus.cc:697] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [term 3 LEADER]: Becoming Leader. State: Replica: 059cb565568c418e80d1a4b2d88a9477, State: Running, Role: LEADER
I20260429 08:00:53.504077 20429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } }
I20260429 08:00:53.505192 20432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059cb565568c418e80d1a4b2d88a9477. Latest consensus state: current_term: 3 leader_uuid: "059cb565568c418e80d1a4b2d88a9477" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:53.505304 20432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:53.505613 20434 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:53.507655 20434 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:53.508045 20434 catalog_manager.cc:1269] Loaded cluster ID: 8aab4ba730d34e47b2f51a81e40442ca
I20260429 08:00:53.508090 20434 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:53.508601 20434 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:53.509042 20434 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Loaded TSK: 0
I20260429 08:00:53.509634 20434 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:00:53.555124 19887 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:53.557611 19887 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:53.560734 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.560445 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35490)
0429 08:00:53.560589 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.560647 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260429 08:00:53.560722 19712 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [term 3 FOLLOWER]: Refusing update from remote peer 059cb565568c418e80d1a4b2d88a9477: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260429 08:00:53.561214 20432 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [LEADER]: Connected to new peer: Peer: permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260429 08:00:53.561481 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.561244 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.0.0.1:35498)
0429 08:00:53.561355 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.561421 (+ 66us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260429 08:00:53.561600 19874 consensus_peers.cc:597] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 -> Peer 41490c9b9bf040a1acee45269d08c82d (127.12.12.190:34475): Couldn't send request to peer 41490c9b9bf040a1acee45269d08c82d. Status: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260429 08:00:53.562754 20436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059cb565568c418e80d1a4b2d88a9477. Latest consensus state: current_term: 3 leader_uuid: "059cb565568c418e80d1a4b2d88a9477" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:53.562868 20436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:53.563921 20432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "059cb565568c418e80d1a4b2d88a9477" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:53.564050 20432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:53.564209 20436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "059cb565568c418e80d1a4b2d88a9477" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41490c9b9bf040a1acee45269d08c82d" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34475 } } peers { permanent_uuid: "177399518fce4172b5943114d5d36fb5" member_type: VOTER last_known_addr { host: "127.12.12.189" port: 45059 } attrs { promote: false } } peers { permanent_uuid: "059cb565568c418e80d1a4b2d88a9477" member_type: VOTER last_known_addr { host: "127.12.12.188" port: 40281 } attrs { promote: false } } }
I20260429 08:00:53.564337 20436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177399518fce4172b5943114d5d36fb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 08:00:53.573774 20430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.570572 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:39375 (local address 127.0.0.1:34568)
0429 08:00:53.570734 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.570749 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:53.570839 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:53.571121 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:53.571124 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:53.571142 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:53.571405 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.571415 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.572511 (+ 1096us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.572515 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:53.573404 (+ 889us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.573412 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.573483 (+ 71us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.573520 (+ 37us) client_negotiation.cc:770] Sending connection context
0429 08:00:53.573559 (+ 39us) client_negotiation.cc:241] Negotiation successful
0429 08:00:53.573615 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260429 08:00:53.574254 20444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.571270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:36893 (local address 127.0.0.1:54208)
0429 08:00:53.571724 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.571738 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:53.571815 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:53.572124 (+ 309us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:53.572127 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:53.572144 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:53.572356 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.572362 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.573136 (+ 774us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.573141 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:53.573929 (+ 788us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.573937 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.574041 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.574054 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:00:53.574101 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 08:00:53.574146 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":68,"threads_started":1}
I20260429 08:00:53.574743 20441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.570574 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.12.12.130:39375)
0429 08:00:53.570852 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:53.570857 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:53.570882 (+ 25us) server_negotiation.cc:408] Connection header received
0429 08:00:53.570945 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:53.570948 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:53.571004 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:53.571516 (+ 512us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:53.571632 (+ 116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.572393 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.573986 (+ 1593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.574487 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.574524 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:53.574578 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 08:00:53.574648 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
I20260429 08:00:53.574776 20442 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.570981 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:35323 (local address 127.0.0.1:41946)
0429 08:00:53.571330 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.571346 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:53.571427 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:53.572176 (+ 749us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:53.572179 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:53.572196 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:53.572392 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.572398 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.573408 (+ 1010us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.573412 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:53.574443 (+ 1031us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.574454 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.574546 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.574559 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:00:53.574604 (+ 45us) client_negotiation.cc:241] Negotiation successful
0429 08:00:53.574667 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260429 08:00:53.575255 20149 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac5bb5df69144e3b428b8e2496913fa (DEFAULT_TABLE table=test-table [id=0a9e6fd46e0a4dfb849eda8759572454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:53.575567 20149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac5bb5df69144e3b428b8e2496913fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:53.575719 20443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.571081 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.12.12.131:35323)
0429 08:00:53.571980 (+ 899us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:53.571984 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:53.571994 (+ 10us) server_negotiation.cc:408] Connection header received
0429 08:00:53.572034 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:53.572036 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:53.572076 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:53.572166 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:53.572515 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.573282 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.574974 (+ 1692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.575442 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.575481 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:53.575538 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 08:00:53.575601 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":73,"threads_started":1}
I20260429 08:00:53.576246 20286 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac5bb5df69144e3b428b8e2496913fa (DEFAULT_TABLE table=test-table [id=0a9e6fd46e0a4dfb849eda8759572454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:53.576508 20286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac5bb5df69144e3b428b8e2496913fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:53.577767 20446 tablet_bootstrap.cc:492] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Bootstrap starting.
I20260429 08:00:53.578302 20445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.571595 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.12.12.129:36893)
0429 08:00:53.571842 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:53.571846 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:53.571858 (+ 12us) server_negotiation.cc:408] Connection header received
0429 08:00:53.571904 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:53.571907 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:53.571951 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:53.572030 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:53.572487 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.573033 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.574491 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.578009 (+ 3518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.578043 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:53.578094 (+ 51us) server_negotiation.cc:300] Negotiation successful
0429 08:00:53.578156 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260429 08:00:53.578810 20446 tablet_bootstrap.cc:654] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:53.578852 20447 tablet_bootstrap.cc:492] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Bootstrap starting.
I20260429 08:00:53.578953 20012 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac5bb5df69144e3b428b8e2496913fa (DEFAULT_TABLE table=test-table [id=0a9e6fd46e0a4dfb849eda8759572454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:53.579209 20012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac5bb5df69144e3b428b8e2496913fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:53.579808 20447 tablet_bootstrap.cc:654] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:53.579950 20446 tablet_bootstrap.cc:492] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: No bootstrap required, opened a new log
I20260429 08:00:53.580042 20446 ts_tablet_manager.cc:1403] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:53.580479 20446 raft_consensus.cc:359] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.580592 20446 raft_consensus.cc:385] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:53.580646 20446 raft_consensus.cc:740] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f597d7c0ffe64506a90069b5bf96ce88, State: Initialized, Role: FOLLOWER
I20260429 08:00:53.580788 20446 consensus_queue.cc:260] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.580994 20447 tablet_bootstrap.cc:492] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: No bootstrap required, opened a new log
I20260429 08:00:53.581130 20447 ts_tablet_manager.cc:1403] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:53.581187 20446 ts_tablet_manager.cc:1434] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:53.582231 20426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.581956 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:55449)
0429 08:00:53.582114 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.582163 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260429 08:00:53.582283 20447 raft_consensus.cc:359] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.582419 20447 raft_consensus.cc:385] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:53.582492 20447 raft_consensus.cc:740] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57838a88843949328cd35e764aa23bb8, State: Initialized, Role: FOLLOWER
I20260429 08:00:53.582640 20447 consensus_queue.cc:260] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.583247 20447 ts_tablet_manager.cc:1434] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:53.583678 20451 tablet_bootstrap.cc:492] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1: Bootstrap starting.
I20260429 08:00:53.583973 20427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.583305 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:34747)
0429 08:00:53.583858 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.583905 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260429 08:00:53.584868 20215 heartbeater.cc:499] Master 127.12.12.188:40281 was elected leader, sending a full tablet report...
I20260429 08:00:53.584972 20352 heartbeater.cc:499] Master 127.12.12.188:40281 was elected leader, sending a full tablet report...
I20260429 08:00:53.585327 20427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.584657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:38319)
0429 08:00:53.585212 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.585263 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260429 08:00:53.585659 20426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.585401 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:41147)
0429 08:00:53.585546 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.585588 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260429 08:00:53.586169 20451 tablet_bootstrap.cc:654] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:53.587460 20451 tablet_bootstrap.cc:492] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1: No bootstrap required, opened a new log
I20260429 08:00:53.587577 20451 ts_tablet_manager.cc:1403] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260429 08:00:53.587982 20451 raft_consensus.cc:359] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.588070 20451 raft_consensus.cc:385] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:53.588129 20451 raft_consensus.cc:740] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad2a529f4a346358315acf2866329d1, State: Initialized, Role: FOLLOWER
I20260429 08:00:53.588286 20451 consensus_queue.cc:260] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.588635 20451 ts_tablet_manager.cc:1434] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:53.589218 20428 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.588983 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:35877)
0429 08:00:53.589091 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.589157 (+ 66us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260429 08:00:53.589994 20079 heartbeater.cc:499] Master 127.12.12.188:40281 was elected leader, sending a full tablet report...
W20260429 08:00:53.590027 20355 tablet.cc:2404] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:53.590445 20428 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.590214 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:54731)
0429 08:00:53.590334 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.590381 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260429 08:00:53.642434 20454 raft_consensus.cc:493] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:53.642580 20454 raft_consensus.cc:515] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.643839 20454 leader_election.cc:290] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375), 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323)
I20260429 08:00:53.647192 20428 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.644014 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:39375 (local address 127.12.12.129:52601)
0429 08:00:53.644145 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.644161 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:53.644276 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:53.644545 (+ 269us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:53.644548 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:53.644588 (+ 40us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:53.644952 (+ 364us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.644963 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.645898 (+ 935us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.645901 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:53.646760 (+ 859us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.646769 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.646894 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.646911 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 08:00:53.646964 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 08:00:53.647034 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260429 08:00:53.647341 20455 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.644110 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:35323 (local address 127.12.12.129:48405)
0429 08:00:53.644506 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.644521 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:53.644625 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:53.644904 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:53.644907 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:53.644923 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:53.645256 (+ 333us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.645263 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.646171 (+ 908us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.646174 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:53.646942 (+ 768us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:53.646953 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.647047 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.647058 (+ 11us) client_negotiation.cc:770] Sending connection context
0429 08:00:53.647105 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 08:00:53.647145 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":120,"threads_started":1}
I20260429 08:00:53.647596 20441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.644109 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:52601 (local address 127.12.12.130:39375)
0429 08:00:53.644286 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:53.644290 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:53.644310 (+ 20us) server_negotiation.cc:408] Connection header received
0429 08:00:53.644383 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:53.644386 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:53.644442 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:53.644535 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:53.645081 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.645747 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.646871 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.647315 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.647376 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:53.647427 (+ 51us) server_negotiation.cc:300] Negotiation successful
0429 08:00:53.647476 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260429 08:00:53.647749 20443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.644290 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:48405 (local address 127.12.12.131:35323)
0429 08:00:53.644467 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:53.644471 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:53.644598 (+ 127us) server_negotiation.cc:408] Connection header received
0429 08:00:53.644737 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:53.644741 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:53.644793 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:53.644895 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:53.645387 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.646038 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.647054 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.647486 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.647515 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:53.647559 (+ 44us) server_negotiation.cc:300] Negotiation successful
0429 08:00:53.647622 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260429 08:00:53.648166 20169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac5bb5df69144e3b428b8e2496913fa" candidate_uuid: "fad2a529f4a346358315acf2866329d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f597d7c0ffe64506a90069b5bf96ce88" is_pre_election: true
I20260429 08:00:53.648202 20306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac5bb5df69144e3b428b8e2496913fa" candidate_uuid: "fad2a529f4a346358315acf2866329d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57838a88843949328cd35e764aa23bb8" is_pre_election: true
I20260429 08:00:53.648316 20169 raft_consensus.cc:2468] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fad2a529f4a346358315acf2866329d1 in term 0.
I20260429 08:00:53.648403 20306 raft_consensus.cc:2468] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fad2a529f4a346358315acf2866329d1 in term 0.
I20260429 08:00:53.648723 19969 leader_election.cc:304] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f597d7c0ffe64506a90069b5bf96ce88, fad2a529f4a346358315acf2866329d1; no voters:
I20260429 08:00:53.648998 20454 raft_consensus.cc:2804] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:53.649062 20454 raft_consensus.cc:493] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:53.649098 20454 raft_consensus.cc:3060] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:53.650007 20454 raft_consensus.cc:515] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.650429 20454 leader_election.cc:290] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [CANDIDATE]: Term 1 election: Requested vote from peers f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375), 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323)
I20260429 08:00:53.650728 20169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac5bb5df69144e3b428b8e2496913fa" candidate_uuid: "fad2a529f4a346358315acf2866329d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f597d7c0ffe64506a90069b5bf96ce88"
I20260429 08:00:53.650759 20306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac5bb5df69144e3b428b8e2496913fa" candidate_uuid: "fad2a529f4a346358315acf2866329d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57838a88843949328cd35e764aa23bb8"
I20260429 08:00:53.650852 20169 raft_consensus.cc:3060] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:53.650852 20306 raft_consensus.cc:3060] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:53.651893 20169 raft_consensus.cc:2468] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fad2a529f4a346358315acf2866329d1 in term 1.
I20260429 08:00:53.651893 20306 raft_consensus.cc:2468] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fad2a529f4a346358315acf2866329d1 in term 1.
I20260429 08:00:53.652156 19968 leader_election.cc:304] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57838a88843949328cd35e764aa23bb8, fad2a529f4a346358315acf2866329d1; no voters:
I20260429 08:00:53.652381 20454 raft_consensus.cc:2804] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:53.652554 20454 raft_consensus.cc:697] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [term 1 LEADER]: Becoming Leader. State: Replica: fad2a529f4a346358315acf2866329d1, State: Running, Role: LEADER
I20260429 08:00:53.652792 20454 consensus_queue.cc:237] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } }
I20260429 08:00:53.653224 20455 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.653019 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:37213)
0429 08:00:53.653114 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.653160 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260429 08:00:53.655031 19887 catalog_manager.cc:5671] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fad2a529f4a346358315acf2866329d1 (127.12.12.129). New cstate: current_term: 1 leader_uuid: "fad2a529f4a346358315acf2866329d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fad2a529f4a346358315acf2866329d1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 36893 } health_report { overall_health: HEALTHY } } }
W20260429 08:00:53.657364 20218 tablet.cc:2404] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:53.672209 20445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.669260 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.12.12.129:36893)
0429 08:00:53.669407 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:53.669411 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:53.669424 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:00:53.669557 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:53.669559 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:53.669610 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:53.669700 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:53.670061 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.670573 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:53.671208 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:53.671408 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:53.671454 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:53.671834 (+ 380us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:53.671945 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:53.672034 (+ 89us) server_negotiation.cc:300] Negotiation successful
0429 08:00:53.672079 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:53.674822 20169 raft_consensus.cc:1275] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Refusing update from remote peer fad2a529f4a346358315acf2866329d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:53.675204 20306 raft_consensus.cc:1275] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Refusing update from remote peer fad2a529f4a346358315acf2866329d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:53.675310 20456 consensus_queue.cc:1048] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f597d7c0ffe64506a90069b5bf96ce88" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:53.675730 20456 consensus_queue.cc:1048] T 8ac5bb5df69144e3b428b8e2496913fa P fad2a529f4a346358315acf2866329d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57838a88843949328cd35e764aa23bb8" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:53.676942 20426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.676650 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:51941)
0429 08:00:53.676818 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.676871 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260429 08:00:53.679245 20460 mvcc.cc:204] Tried to move back new op lower bound from 7280433781447110656 to 7280433781364408320. Current Snapshot: MvccSnapshot[applied={T|T < 7280433781447110656}]
I20260429 08:00:53.679519 20427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.678709 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:40009)
0429 08:00:53.679386 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.679439 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585}
I20260429 08:00:53.680558 20455 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.679891 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.129:58547)
0429 08:00:53.680439 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.680485 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
I20260429 08:00:53.681253 20427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.681006 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.131:59441)
0429 08:00:53.681148 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.681191 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260429 08:00:53.681253 20426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.681004 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34475 (local address 127.12.12.130:35351)
0429 08:00:53.681155 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.681191 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:34475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260429 08:00:53.683847 20463 mvcc.cc:204] Tried to move back new op lower bound from 7280433781447110656 to 7280433781364408320. Current Snapshot: MvccSnapshot[applied={T|T < 7280433781447110656}]
I20260429 08:00:53.688796 19887 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:53.688985 19887 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:53.693114 19887 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 059cb565568c418e80d1a4b2d88a9477: Sending DeleteTablet for 3 replicas of tablet 8ac5bb5df69144e3b428b8e2496913fa
I20260429 08:00:53.693686 20149 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ac5bb5df69144e3b428b8e2496913fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:53 UTC) from {username='oryx', principal='oryx/127.12.12.188@KRBTEST.COM'} at 127.0.0.1:34568
I20260429 08:00:53.694010 20286 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ac5bb5df69144e3b428b8e2496913fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:53 UTC) from {username='oryx', principal='oryx/127.12.12.188@KRBTEST.COM'} at 127.0.0.1:41946
I20260429 08:00:53.694499 20469 tablet_replica.cc:333] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: stopping tablet replica
I20260429 08:00:53.694509 20012 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ac5bb5df69144e3b428b8e2496913fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:53 UTC) from {username='oryx', principal='oryx/127.12.12.188@KRBTEST.COM'} at 127.0.0.1:54208
I20260429 08:00:53.694741 20470 tablet_replica.cc:333] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: stopping tablet replica
I20260429 08:00:53.695482 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19950
I20260429 08:00:53.697669 20470 raft_consensus.cc:2243] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:53.697803 20470 raft_consensus.cc:2272] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:53.697939 20469 raft_consensus.cc:2243] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:53.698132 20469 raft_consensus.cc:2272] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:53.698554 20470 ts_tablet_manager.cc:1916] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:53.698834 20469 ts_tablet_manager.cc:1916] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:53.700512 20469 ts_tablet_manager.cc:1929] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:53.700611 20469 log.cc:1199] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/8ac5bb5df69144e3b428b8e2496913fa
I20260429 08:00:53.700964 20469 ts_tablet_manager.cc:1950] T 8ac5bb5df69144e3b428b8e2496913fa P f597d7c0ffe64506a90069b5bf96ce88: Deleting consensus metadata
I20260429 08:00:53.701474 20470 ts_tablet_manager.cc:1929] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:53.701555 20470 log.cc:1199] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/8ac5bb5df69144e3b428b8e2496913fa
I20260429 08:00:53.701613 19874 catalog_manager.cc:5002] TS f597d7c0ffe64506a90069b5bf96ce88 (127.12.12.130:39375): tablet 8ac5bb5df69144e3b428b8e2496913fa (table test-table [id=0a9e6fd46e0a4dfb849eda8759572454]) successfully deleted
I20260429 08:00:53.701855 20470 ts_tablet_manager.cc:1950] T 8ac5bb5df69144e3b428b8e2496913fa P 57838a88843949328cd35e764aa23bb8: Deleting consensus metadata
I20260429 08:00:53.702538 19873 catalog_manager.cc:5002] TS 57838a88843949328cd35e764aa23bb8 (127.12.12.131:35323): tablet 8ac5bb5df69144e3b428b8e2496913fa (table test-table [id=0a9e6fd46e0a4dfb849eda8759572454]) successfully deleted
I20260429 08:00:53.704308 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 20086
I20260429 08:00:53.704617 20442 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:53.704338 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:36893 (local address 127.0.0.1:54222)
0429 08:00:53.704478 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:53.704530 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:36893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260429 08:00:53.704963 19873 catalog_manager.cc:4729] TS fad2a529f4a346358315acf2866329d1 (127.12.12.129:36893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ac5bb5df69144e3b428b8e2496913fa: Network error: Client connection negotiation failed: client connection to 127.12.12.129:36893: connect: Connection refused (error 111)
I20260429 08:00:53.710271 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 20222
I20260429 08:00:53.716915 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19564
I20260429 08:00:53.722887 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 19674
2026-04-29T08:00:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8889 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:53 dist-test-slave-nhqz krb5kdc[20474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:53 dist-test-slave-nhqz krb5kdc[20474](info): set up 2 sockets
Apr 29 08:00:53 dist-test-slave-nhqz krb5kdc[20474](info): commencing operation
krb5kdc: starting...
W20260429 08:00:55.714217 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.967s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:55 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:55.848433 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34411
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:55.947360 20490 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:55.947635 20490 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:55.947700 20490 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:55.950965 20490 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:55.951021 20490 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 08:00:55.951038 20490 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:55.951059 20490 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:55.951076 20490 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:55.951093 20490 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:55.955147 20490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:55.956138 20490 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:55.957019 20490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:55.962335 20498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:55.962430 20496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:55.962363 20495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:55.962606 20490 server_base.cc:1061] running on GCE node
I20260429 08:00:55.963121 20490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:55.963999 20490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:55.965173 20490 hybrid_clock.cc:648] HybridClock initialized: now 1777449655965155 us; error 36 us; skew 500 ppm
Apr 29 08:00:55 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449655, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:55.967621 20490 init.cc:377] Logged in from keytab as oryx/127.12.12.190@KRBTEST.COM (short username oryx)
I20260429 08:00:55.968616 20490 webserver.cc:492] Webserver started at http://127.12.12.190:40691/ using document root <none> and password file <none>
I20260429 08:00:55.969169 20490 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:55.969218 20490 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:55.969424 20490 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:55.971010 20490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "ab69a0bff21940ca86f1f7a034fe8132"
format_stamp: "Formatted at 2026-04-29 08:00:55 on dist-test-slave-nhqz"
server_key: "605d7a5008063831bdd0e6be555bc733"
server_key_iv: "dfc40346ea399772c8aedcb96133d21d"
server_key_version: "encryptionkey@0"
I20260429 08:00:55.971474 20490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "ab69a0bff21940ca86f1f7a034fe8132"
format_stamp: "Formatted at 2026-04-29 08:00:55 on dist-test-slave-nhqz"
server_key: "605d7a5008063831bdd0e6be555bc733"
server_key_iv: "dfc40346ea399772c8aedcb96133d21d"
server_key_version: "encryptionkey@0"
I20260429 08:00:55.974601 20490 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:00:55.976778 20505 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:55.977787 20490 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:00:55.977912 20490 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "ab69a0bff21940ca86f1f7a034fe8132"
format_stamp: "Formatted at 2026-04-29 08:00:55 on dist-test-slave-nhqz"
server_key: "605d7a5008063831bdd0e6be555bc733"
server_key_iv: "dfc40346ea399772c8aedcb96133d21d"
server_key_version: "encryptionkey@0"
I20260429 08:00:55.978021 20490 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:55.997099 20490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:56.000429 20490 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:56.000643 20490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:56.009434 20490 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:34411
I20260429 08:00:56.009653 20567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:34411 every 8 connection(s)
I20260429 08:00:56.010483 20490 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:00:56.013127 20568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.013825 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 20490
I20260429 08:00:56.013928 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:00:56.014245 12338 external_mini_cluster.cc:1468] Setting key 4a77507a222c121b97facc947f71ed19
I20260429 08:00:56.018394 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Bootstrap starting.
I20260429 08:00:56.020569 20568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.021627 20568 log.cc:826] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Log is configured to *not* fsync() on all Append() calls
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:56.023950 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: No bootstrap required, opened a new log
I20260429 08:00:56.027213 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.015623 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.12.12.190:34411)
0429 08:00:56.016064 (+ 441us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.016074 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.016107 (+ 33us) server_negotiation.cc:408] Connection header received
0429 08:00:56.016625 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.016643 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.016993 (+ 350us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.017302 (+ 309us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.018198 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.019002 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.019705 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.019998 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.022783 (+ 2785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.022800 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.022813 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.022839 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.024741 (+ 1902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.025266 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.025270 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.025274 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.025344 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.025619 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.025621 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.025623 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.025920 (+ 297us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.026055 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.026402 (+ 347us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.026617 (+ 215us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260429 08:00:56.027828 20568 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab69a0bff21940ca86f1f7a034fe8132" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34411 } }
I20260429 08:00:56.028004 20568 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.028075 20568 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab69a0bff21940ca86f1f7a034fe8132, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.028676 20568 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab69a0bff21940ca86f1f7a034fe8132" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34411 } }
I20260429 08:00:56.028870 20568 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:00:56.028949 20568 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:00:56.029038 20568 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.029917 20568 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab69a0bff21940ca86f1f7a034fe8132" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34411 } }
I20260429 08:00:56.030200 20568 leader_election.cc:304] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab69a0bff21940ca86f1f7a034fe8132; no voters:
I20260429 08:00:56.030472 20568 leader_election.cc:290] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:00:56.030622 20573 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:56.030830 20573 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [term 1 LEADER]: Becoming Leader. State: Replica: ab69a0bff21940ca86f1f7a034fe8132, State: Running, Role: LEADER
I20260429 08:00:56.031435 20573 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab69a0bff21940ca86f1f7a034fe8132" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34411 } }
I20260429 08:00:56.031632 20568 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:00:56.033073 20575 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab69a0bff21940ca86f1f7a034fe8132. Latest consensus state: current_term: 1 leader_uuid: "ab69a0bff21940ca86f1f7a034fe8132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab69a0bff21940ca86f1f7a034fe8132" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34411 } } }
I20260429 08:00:56.033062 20574 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab69a0bff21940ca86f1f7a034fe8132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab69a0bff21940ca86f1f7a034fe8132" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34411 } } }
I20260429 08:00:56.033193 20575 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:56.033193 20574 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132 [sys.catalog]: This master's current role is: LEADER
I20260429 08:00:56.033593 20585 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:00:56.035970 20585 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:00:56.041106 20585 catalog_manager.cc:1357] Generated new cluster ID: 6917ad52a600443391fcbbe80f89b939
I20260429 08:00:56.041174 20585 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:00:56.049795 20585 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:00:56.050638 20585 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:00:56.058732 20585 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Generated new TSK 0
I20260429 08:00:56.059238 20585 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:56.109275 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34411
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:56.213620 20596 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:56.213825 20596 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:56.213927 20596 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:56.217209 20596 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:56.217269 20596 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:56.217365 20596 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 08:00:56.217464 20596 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:00:56.221547 20596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:34411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:56.222576 20596 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:56.223439 20596 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:56.229827 20601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:56.229962 20604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:56.229851 20602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:56.230015 20596 server_base.cc:1061] running on GCE node
I20260429 08:00:56.230463 20596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:56.231011 20596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:56.232230 20596 hybrid_clock.cc:648] HybridClock initialized: now 1777449656232209 us; error 40 us; skew 500 ppm
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:56.234905 20596 init.cc:377] Logged in from keytab as oryx/127.12.12.129@KRBTEST.COM (short username oryx)
I20260429 08:00:56.235816 20596 webserver.cc:492] Webserver started at http://127.12.12.129:37685/ using document root <none> and password file <none>
I20260429 08:00:56.236301 20596 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:56.236356 20596 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:56.236501 20596 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:56.238147 20596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "195cfbba0a504bc1a0b9d3c05e5c6026"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "1f54d7a143b94484740dd9f68cdd3bf1"
server_key_iv: "b65bfb7d8ac878d47beaed98f1d581a3"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.238571 20596 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "195cfbba0a504bc1a0b9d3c05e5c6026"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "1f54d7a143b94484740dd9f68cdd3bf1"
server_key_iv: "b65bfb7d8ac878d47beaed98f1d581a3"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.241703 20596 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260429 08:00:56.243916 20611 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:56.244938 20596 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:00:56.245031 20596 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "195cfbba0a504bc1a0b9d3c05e5c6026"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "1f54d7a143b94484740dd9f68cdd3bf1"
server_key_iv: "b65bfb7d8ac878d47beaed98f1d581a3"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.245105 20596 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:56.253115 20596 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:56.255441 20596 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:56.255580 20596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:56.256912 20596 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:56.256983 20596 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:56.257059 20596 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:56.257113 20596 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:56.268242 20596 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:42269
I20260429 08:00:56.268292 20726 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:42269 every 8 connection(s)
I20260429 08:00:56.269164 20596 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:00:56.269182 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.257492 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:49885 (local address 127.12.12.190:34411)
0429 08:00:56.257645 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.257650 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.258366 (+ 716us) server_negotiation.cc:408] Connection header received
0429 08:00:56.259195 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.259200 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.259266 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.259379 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.260735 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.261597 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.262423 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.262648 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.265261 (+ 2613us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.265287 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.265293 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.265327 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.266854 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.267348 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.267352 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.267354 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.267411 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.267696 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.267700 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.267701 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.267839 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.267948 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.268931 (+ 983us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.269051 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 08:00:56.269271 20620 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.257782 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.12.12.129:49885)
0429 08:00:56.258210 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.258244 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.259002 (+ 758us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.259523 (+ 521us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.259530 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.259957 (+ 427us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:56.260576 (+ 619us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.260592 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.261728 (+ 1136us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.261733 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.262330 (+ 597us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.262337 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.262518 (+ 181us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.263147 (+ 629us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:56.263174 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:56.265110 (+ 1936us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:56.266971 (+ 1861us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.266978 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.266988 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.267247 (+ 259us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.267507 (+ 260us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.267511 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.267513 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.267602 (+ 89us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.267942 (+ 340us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:56.267947 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:56.268183 (+ 236us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.268405 (+ 222us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.268585 (+ 180us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20260429 08:00:56.270642 20727 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34411
I20260429 08:00:56.270843 20727 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:56.271353 20727 heartbeater.cc:507] Master 127.12.12.190:34411 requested a full tablet report, sending...
I20260429 08:00:56.272921 20522 ts_manager.cc:194] Registered new tserver with Master: 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269)
I20260429 08:00:56.274037 20522 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:49885
I20260429 08:00:56.274950 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 20596
I20260429 08:00:56.275068 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:00:56.275321 12338 external_mini_cluster.cc:1468] Setting key 357efd8b69936eae5e27f3dca6f711db
I20260429 08:00:56.279088 20733 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.272604 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.12.12.129:56715)
0429 08:00:56.272960 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.272971 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.273093 (+ 122us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.273358 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.273361 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.273522 (+ 161us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:56.273723 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.273728 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.274442 (+ 714us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.274447 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.274939 (+ 492us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.274946 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.275067 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.275634 (+ 567us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:56.275673 (+ 39us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:56.276006 (+ 333us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:56.277714 (+ 1708us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.277721 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.277725 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.277948 (+ 223us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.278238 (+ 290us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.278240 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.278242 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.278285 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.278694 (+ 409us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:56.278697 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:56.278759 (+ 62us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.278882 (+ 123us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.278966 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
I20260429 08:00:56.279280 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.272667 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:56715 (local address 127.12.12.190:34411)
0429 08:00:56.272834 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.272840 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.273048 (+ 208us) server_negotiation.cc:408] Connection header received
0429 08:00:56.273204 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.273208 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.273255 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.273345 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.273829 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.274332 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.275051 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.275253 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.276117 (+ 864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.276133 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.276136 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.276165 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.277607 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.278059 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.278064 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.278066 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.278121 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.278415 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.278420 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.278423 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.278597 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.278694 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.278960 (+ 266us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.279069 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:56.322204 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34411
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:56.423460 20738 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:56.423730 20738 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:56.423807 20738 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:56.427323 20738 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:56.427378 20738 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:56.427428 20738 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 08:00:56.427469 20738 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:00:56.431314 20738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:34411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:56.432267 20738 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:56.433113 20738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:56.439266 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:56.439311 20738 server_base.cc:1061] running on GCE node
W20260429 08:00:56.439266 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:56.439263 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:56.439889 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:56.440445 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:56.441607 20738 hybrid_clock.cc:648] HybridClock initialized: now 1777449656441575 us; error 45 us; skew 500 ppm
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:56.444057 20738 init.cc:377] Logged in from keytab as oryx/127.12.12.130@KRBTEST.COM (short username oryx)
I20260429 08:00:56.445092 20738 webserver.cc:492] Webserver started at http://127.12.12.130:33731/ using document root <none> and password file <none>
I20260429 08:00:56.445564 20738 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:56.445608 20738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:56.445762 20738 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:56.447381 20738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "7ff78689746743a7be7900a6c3cfc61c"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "3a83b69582f26a62dc22308d545648f4"
server_key_iv: "e103bb989f09f8eb30c4091a5bdc9d47"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.447795 20738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "7ff78689746743a7be7900a6c3cfc61c"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "3a83b69582f26a62dc22308d545648f4"
server_key_iv: "e103bb989f09f8eb30c4091a5bdc9d47"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.450992 20738 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260429 08:00:56.452890 20753 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:56.453786 20738 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:56.453896 20738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "7ff78689746743a7be7900a6c3cfc61c"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "3a83b69582f26a62dc22308d545648f4"
server_key_iv: "e103bb989f09f8eb30c4091a5bdc9d47"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.453984 20738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:56.473860 20738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:56.476948 20738 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:56.477149 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:56.478668 20738 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:56.478749 20738 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:56.478814 20738 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:56.478859 20738 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:56.491143 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.479167 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:51605 (local address 127.12.12.190:34411)
0429 08:00:56.479317 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.479323 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.480133 (+ 810us) server_negotiation.cc:408] Connection header received
0429 08:00:56.481443 (+ 1310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.481446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.481497 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.481580 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.483422 (+ 1842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.483996 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.484625 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.484803 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.487710 (+ 2907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.487737 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.487745 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.487770 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.489299 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.489798 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.489803 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.489805 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.489857 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.490160 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.490164 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.490166 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.490309 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.490422 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.490867 (+ 445us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.490985 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 08:00:56.491750 20738 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:33611
I20260429 08:00:56.491775 20868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:33611 every 8 connection(s)
I20260429 08:00:56.491796 20762 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.479491 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.12.12.130:51605)
0429 08:00:56.479959 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.480006 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.481187 (+ 1181us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.481817 (+ 630us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.481831 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.482499 (+ 668us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:56.483219 (+ 720us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.483258 (+ 39us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.484122 (+ 864us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.484126 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.484508 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.484515 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.484729 (+ 214us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.485346 (+ 617us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:56.485376 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:56.487505 (+ 2129us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:56.489453 (+ 1948us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.489459 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.489470 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.489697 (+ 227us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.489970 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.489973 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.489974 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.490069 (+ 95us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.490415 (+ 346us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:56.490420 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:56.490624 (+ 204us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.490820 (+ 196us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.491054 (+ 234us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260429 08:00:56.492725 20738 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:00:56.494309 20869 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34411
I20260429 08:00:56.495568 20869 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:56.496040 20869 heartbeater.cc:507] Master 127.12.12.190:34411 requested a full tablet report, sending...
I20260429 08:00:56.497154 20522 ts_manager.cc:194] Registered new tserver with Master: 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611)
I20260429 08:00:56.497633 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 20738
I20260429 08:00:56.497763 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:00:56.497774 20522 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:51605
I20260429 08:00:56.498072 12338 external_mini_cluster.cc:1468] Setting key 10a99cbfa8d84048f6081aa77e7c62de
I20260429 08:00:56.502353 20875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.495193 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.12.12.130:33061)
0429 08:00:56.495610 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.495621 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.495730 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.495972 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.495975 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.496184 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:56.496399 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.496405 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.497407 (+ 1002us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.497412 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.498003 (+ 591us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.498016 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.498151 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.498767 (+ 616us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:56.498782 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:56.499134 (+ 352us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:56.500780 (+ 1646us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.500787 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.500792 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.501085 (+ 293us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.501373 (+ 288us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.501377 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.501380 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.501436 (+ 56us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.501901 (+ 465us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:56.501904 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:56.501993 (+ 89us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.502115 (+ 122us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.502211 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":103,"threads_started":1}
I20260429 08:00:56.502799 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.495276 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:33061 (local address 127.12.12.190:34411)
0429 08:00:56.495472 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.495476 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.495691 (+ 215us) server_negotiation.cc:408] Connection header received
0429 08:00:56.495829 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.495832 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.495873 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.495959 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.496521 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.497264 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.498132 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.498305 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.499266 (+ 961us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.499281 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.499284 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.499308 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.500665 (+ 1357us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.501196 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.501202 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.501205 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.501261 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.501534 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.501539 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.501542 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.501771 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.501891 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.502198 (+ 307us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.502316 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for oryx/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab.
I20260429 08:00:56.544679 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34411
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:56.641048 20880 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:56.641247 20880 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:56.641326 20880 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:56.644440 20880 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:56.644537 20880 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:56.644636 20880 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 08:00:56.644721 20880 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:00:56.648766 20880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/krb5kdc/oryx.keytab
--principal=oryx/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.12.190:34411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:56.649819 20880 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:56.650604 20880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:56.656908 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:56.656914 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:56.656976 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:56.657159 20880 server_base.cc:1061] running on GCE node
I20260429 08:00:56.657495 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:56.658058 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:56.659232 20880 hybrid_clock.cc:648] HybridClock initialized: now 1777449656659217 us; error 41 us; skew 500 ppm
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:00:56.661965 20880 init.cc:377] Logged in from keytab as oryx/127.12.12.131@KRBTEST.COM (short username oryx)
I20260429 08:00:56.662932 20880 webserver.cc:492] Webserver started at http://127.12.12.131:42377/ using document root <none> and password file <none>
I20260429 08:00:56.663443 20880 fs_manager.cc:362] Metadata directory not provided
I20260429 08:00:56.663488 20880 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:00:56.663637 20880 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:00:56.665316 20880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "05b46d250f956f47a49e0f3ff88ff176"
server_key_iv: "0b72ea0f1d15c3057504184aa800194f"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.665737 20880 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "05b46d250f956f47a49e0f3ff88ff176"
server_key_iv: "0b72ea0f1d15c3057504184aa800194f"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.668746 20880 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:00:56.670910 20895 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:56.671933 20880 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:56.672045 20880 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
format_stamp: "Formatted at 2026-04-29 08:00:56 on dist-test-slave-nhqz"
server_key: "05b46d250f956f47a49e0f3ff88ff176"
server_key_iv: "0b72ea0f1d15c3057504184aa800194f"
server_key_version: "encryptionkey@0"
I20260429 08:00:56.672133 20880 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:00:56.684629 20880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:00:56.687427 20880 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:00:56.687577 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:00:56.689090 20880 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:00:56.689139 20880 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:00:56.689208 20880 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:00:56.689239 20880 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:56.700892 20880 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:35441
I20260429 08:00:56.700927 21010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:35441 every 8 connection(s)
I20260429 08:00:56.701519 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.689692 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:41881 (local address 127.12.12.190:34411)
0429 08:00:56.689840 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.689844 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.690565 (+ 721us) server_negotiation.cc:408] Connection header received
0429 08:00:56.691436 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.691439 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.691509 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.691603 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.693213 (+ 1610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.693872 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.694815 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.695055 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.697664 (+ 2609us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.697686 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.697689 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.697718 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.699399 (+ 1681us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.700019 (+ 620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.700023 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.700025 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.700078 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.700373 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.700376 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.700378 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.700530 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.700644 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.701255 (+ 611us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.701362 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:00:56.701817 20880 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:00:56.702097 20904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.690033 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.12.12.131:41881)
0429 08:00:56.690436 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.690466 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.691216 (+ 750us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.691891 (+ 675us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.691898 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.692498 (+ 600us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:56.693068 (+ 570us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.693080 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.693991 (+ 911us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.693996 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.694665 (+ 669us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.694673 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.694911 (+ 238us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.695516 (+ 605us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:56.695544 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:56.697477 (+ 1933us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:56.699526 (+ 2049us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.699533 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.699547 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.699897 (+ 350us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.700173 (+ 276us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.700177 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.700179 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.700275 (+ 96us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.700725 (+ 450us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:56.700732 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:56.700999 (+ 267us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.701207 (+ 208us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.701424 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260429 08:00:56.703454 21011 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34411
I20260429 08:00:56.703593 21011 heartbeater.cc:461] Registering TS with master...
I20260429 08:00:56.704111 21011 heartbeater.cc:507] Master 127.12.12.190:34411 requested a full tablet report, sending...
I20260429 08:00:56.705186 20522 ts_manager.cc:194] Registered new tserver with Master: b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:56.705834 20522 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:41881
I20260429 08:00:56.710280 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 20880
I20260429 08:00:56.710373 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:00:56.710647 12338 external_mini_cluster.cc:1468] Setting key 2f9e470f25bf456d8eb42515d2a5db5c
I20260429 08:00:56.711941 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.705680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:42823 (local address 127.12.12.190:34411)
0429 08:00:56.705819 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.705823 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.706199 (+ 376us) server_negotiation.cc:408] Connection header received
0429 08:00:56.706303 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.706306 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.706350 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.706418 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.707009 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.707499 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.708070 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.708189 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.709000 (+ 811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.709013 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.709015 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.709036 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.710493 (+ 1457us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.710909 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.710912 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.710914 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.710963 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.711236 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.711239 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.711241 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.711383 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.711489 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.711723 (+ 234us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.711814 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:00:56.711962 21017 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.705608 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.12.12.131:42823)
0429 08:00:56.706138 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.706149 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.706228 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.706452 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.706455 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.706634 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:56.706898 (+ 264us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.706905 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.707622 (+ 717us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.707625 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.707979 (+ 354us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.707985 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.708072 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.708531 (+ 459us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:56.708542 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:56.708886 (+ 344us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:56.710592 (+ 1706us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.710597 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.710599 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.710804 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.711078 (+ 274us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:56.711081 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:56.711083 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:56.711144 (+ 61us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:56.711480 (+ 336us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:56.711484 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:56.711559 (+ 75us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.711670 (+ 111us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.711788 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":99,"threads_started":1}
I20260429 08:00:56.712668 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260429 08:00:56.733740 20571 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:56.728436 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.12.12.190:34411)
0429 08:00:56.728569 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.728573 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.728584 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:00:56.728624 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.728626 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.728679 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.728748 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.729529 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.730005 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.730600 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.730721 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.733563 (+ 2842us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34046: BlockingRecv error: recv got EOF from 127.0.0.1:34046 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Apr 29 08:00:56 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:56.741859 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.735058 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.12.12.190:34411)
0429 08:00:56.735178 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.735186 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.735293 (+ 107us) server_negotiation.cc:408] Connection header received
0429 08:00:56.735378 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.735380 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.735417 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.735483 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:56.736072 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.736510 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.737078 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.737226 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.738969 (+ 1743us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:56.738990 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:56.738992 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:56.739014 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:56.740355 (+ 1341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.740749 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.740753 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.740757 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.740875 (+ 118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:56.741116 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:56.741120 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:56.741122 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:56.741251 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:56.741382 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.741602 (+ 220us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.741704 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 08:00:56.744242 20521 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:56.746737 20521 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:56.759037 21033 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.754260 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:35441 (local address 127.0.0.1:39940)
0429 08:00:56.754523 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.754556 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.754685 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.755877 (+ 1192us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.755886 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.755912 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.756202 (+ 290us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.756208 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.757838 (+ 1630us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.757842 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.758679 (+ 837us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.758686 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.758777 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.758795 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.758839 (+ 44us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.758879 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260429 08:00:56.759598 21034 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.754441 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.12.12.131:35441)
0429 08:00:56.755418 (+ 977us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.755426 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.755445 (+ 19us) server_negotiation.cc:408] Connection header received
0429 08:00:56.755527 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.755532 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.755707 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.756335 (+ 628us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.756489 (+ 154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.757744 (+ 1255us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.758821 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.759249 (+ 428us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.759370 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.759439 (+ 69us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.759501 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":98,"threads_started":1}
I20260429 08:00:56.759922 21037 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.754979 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.12.12.129:42269)
0429 08:00:56.755222 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.755228 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.755450 (+ 222us) server_negotiation.cc:408] Connection header received
0429 08:00:56.755527 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.755532 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.755712 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.755850 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.756370 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.757450 (+ 1080us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.758953 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.759519 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.759609 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.759678 (+ 69us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.759756 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260429 08:00:56.760187 21035 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.754809 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:42269 (local address 127.0.0.1:45920)
0429 08:00:56.755248 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.755263 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.755356 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.756018 (+ 662us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.756021 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.756036 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.756253 (+ 217us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.756259 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.757624 (+ 1365us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.757627 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.758549 (+ 922us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.758558 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.758666 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.758715 (+ 49us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.758799 (+ 84us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.758872 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":129,"threads_started":1}
I20260429 08:00:56.761817 21036 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.754880 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:33611 (local address 127.0.0.1:49594)
0429 08:00:56.755534 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.755546 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.755626 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.756134 (+ 508us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.756137 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.756168 (+ 31us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.756410 (+ 242us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.756419 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.759341 (+ 2922us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.759345 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.760174 (+ 829us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.760183 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.760286 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.761581 (+ 1295us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.761655 (+ 74us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.761708 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":112,"threads_started":1}
I20260429 08:00:56.762290 21038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.755065 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.12.12.130:33611)
0429 08:00:56.755363 (+ 298us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.755369 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.755730 (+ 361us) server_negotiation.cc:408] Connection header received
0429 08:00:56.755819 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.755826 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.755994 (+ 168us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.756146 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.756539 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.757744 (+ 1205us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.760923 (+ 3179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.761452 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.761539 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.762073 (+ 534us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.762161 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
I20260429 08:00:56.762544 20945 tablet_service.cc:1511] Processing CreateTablet for tablet 2945cd938f7543168891790e5509370a (DEFAULT_TABLE table=test-table [id=4f68e1785bb8427f95bb4b6c18888541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:56.762799 20661 tablet_service.cc:1511] Processing CreateTablet for tablet 2945cd938f7543168891790e5509370a (DEFAULT_TABLE table=test-table [id=4f68e1785bb8427f95bb4b6c18888541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:56.763438 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2945cd938f7543168891790e5509370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.763609 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2945cd938f7543168891790e5509370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.768406 21040 tablet_bootstrap.cc:492] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Bootstrap starting.
I20260429 08:00:56.768882 21039 tablet_bootstrap.cc:492] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Bootstrap starting.
I20260429 08:00:56.770614 21039 tablet_bootstrap.cc:654] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.771341 21039 log.cc:826] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:56.771358 20802 tablet_service.cc:1511] Processing CreateTablet for tablet 2945cd938f7543168891790e5509370a (DEFAULT_TABLE table=test-table [id=4f68e1785bb8427f95bb4b6c18888541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:56.772205 20802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2945cd938f7543168891790e5509370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.772753 21040 tablet_bootstrap.cc:654] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.773137 21039 tablet_bootstrap.cc:492] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: No bootstrap required, opened a new log
I20260429 08:00:56.773325 21039 ts_tablet_manager.cc:1403] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:00:56.773797 21040 log.cc:826] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:56.775184 21040 tablet_bootstrap.cc:492] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: No bootstrap required, opened a new log
I20260429 08:00:56.775369 21040 ts_tablet_manager.cc:1403] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260429 08:00:56.777655 21040 raft_consensus.cc:359] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.777709 21039 raft_consensus.cc:359] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.777861 21040 raft_consensus.cc:385] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.777894 21039 raft_consensus.cc:385] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.777910 21040 raft_consensus.cc:740] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b0969a9fd04f3aa4e5a27e0d21e31b, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.777943 21039 raft_consensus.cc:740] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.778292 21043 tablet_bootstrap.cc:492] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Bootstrap starting.
I20260429 08:00:56.778324 21039 consensus_queue.cc:260] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.778378 21040 consensus_queue.cc:260] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.778931 20727 heartbeater.cc:499] Master 127.12.12.190:34411 was elected leader, sending a full tablet report...
I20260429 08:00:56.779126 21011 heartbeater.cc:499] Master 127.12.12.190:34411 was elected leader, sending a full tablet report...
I20260429 08:00:56.779153 21039 ts_tablet_manager.cc:1434] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260429 08:00:56.779986 21040 ts_tablet_manager.cc:1434] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 08:00:56.781327 21043 tablet_bootstrap.cc:654] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.783075 21043 log.cc:826] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Log is configured to *not* fsync() on all Append() calls
I20260429 08:00:56.784648 21043 tablet_bootstrap.cc:492] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: No bootstrap required, opened a new log
I20260429 08:00:56.784890 21043 ts_tablet_manager.cc:1403] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260429 08:00:56.787042 21043 raft_consensus.cc:359] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.787256 21043 raft_consensus.cc:385] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.787333 21043 raft_consensus.cc:740] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.787535 21044 raft_consensus.cc:493] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:56.787655 21044 raft_consensus.cc:515] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.787770 21043 consensus_queue.cc:260] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.788327 20869 heartbeater.cc:499] Master 127.12.12.190:34411 was elected leader, sending a full tablet report...
I20260429 08:00:56.788556 21043 ts_tablet_manager.cc:1434] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260429 08:00:56.788662 21044 leader_election.cc:290] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441), 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611)
I20260429 08:00:56.792485 21049 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.789197 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:33611 (local address 127.12.12.129:34291)
0429 08:00:56.789537 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.789549 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.789688 (+ 139us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.789922 (+ 234us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.789925 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.789941 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.790210 (+ 269us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.790218 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.791230 (+ 1012us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.791233 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.792109 (+ 876us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.792118 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.792221 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.792240 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.792289 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.792339 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":71,"threads_started":1}
I20260429 08:00:56.792758 21038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.789360 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:34291 (local address 127.12.12.130:33611)
0429 08:00:56.789518 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.789522 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.789636 (+ 114us) server_negotiation.cc:408] Connection header received
0429 08:00:56.789791 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.789800 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.789835 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.789912 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.790327 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.790946 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.792205 (+ 1259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.792546 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.792580 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.792621 (+ 41us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.792663 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 08:00:56.793097 21034 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.788955 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:37413 (local address 127.12.12.131:35441)
0429 08:00:56.789093 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.789098 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.789180 (+ 82us) server_negotiation.cc:408] Connection header received
0429 08:00:56.789384 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.789387 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.789438 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.789489 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.790035 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.790711 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.791909 (+ 1198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.792359 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.792413 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.792919 (+ 506us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.792983 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 08:00:56.793256 20823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2945cd938f7543168891790e5509370a" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff78689746743a7be7900a6c3cfc61c" is_pre_election: true
I20260429 08:00:56.793494 21048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.788872 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:35441 (local address 127.12.12.129:37413)
0429 08:00:56.789092 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.789107 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.789265 (+ 158us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.789626 (+ 361us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.789629 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.789647 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.789914 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.789922 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.790835 (+ 913us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.790839 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.791686 (+ 847us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.791695 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.792463 (+ 768us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.792475 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.793287 (+ 812us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.793363 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260429 08:00:56.793493 20823 raft_consensus.cc:2468] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 0.
I20260429 08:00:56.793843 20615 leader_election.cc:304] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195cfbba0a504bc1a0b9d3c05e5c6026, 7ff78689746743a7be7900a6c3cfc61c; no voters:
I20260429 08:00:56.794063 21044 raft_consensus.cc:2804] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:56.794134 21044 raft_consensus.cc:493] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:56.794186 21044 raft_consensus.cc:3060] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.794132 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2945cd938f7543168891790e5509370a" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" is_pre_election: true
I20260429 08:00:56.794378 20965 raft_consensus.cc:2468] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 0.
I20260429 08:00:56.795226 21044 raft_consensus.cc:515] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.795584 21044 leader_election.cc:290] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 election: Requested vote from peers b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441), 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611)
I20260429 08:00:56.795881 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2945cd938f7543168891790e5509370a" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
I20260429 08:00:56.795938 20823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2945cd938f7543168891790e5509370a" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff78689746743a7be7900a6c3cfc61c"
I20260429 08:00:56.795975 20965 raft_consensus.cc:3060] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.796089 20823 raft_consensus.cc:3060] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.796989 20965 raft_consensus.cc:2468] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 1.
I20260429 08:00:56.797026 20823 raft_consensus.cc:2468] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 1.
I20260429 08:00:56.797314 20615 leader_election.cc:304] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195cfbba0a504bc1a0b9d3c05e5c6026, b9b0969a9fd04f3aa4e5a27e0d21e31b; no voters:
I20260429 08:00:56.797482 21044 raft_consensus.cc:2804] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:56.797700 21044 raft_consensus.cc:697] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 LEADER]: Becoming Leader. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Running, Role: LEADER
I20260429 08:00:56.797917 21044 consensus_queue.cc:237] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } }
I20260429 08:00:56.800864 20522 catalog_manager.cc:5671] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 reported cstate change: term changed from 0 to 1, leader changed from <none> to 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129). New cstate: current_term: 1 leader_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } health_report { overall_health: HEALTHY } } }
I20260429 08:00:56.821379 21037 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.818481 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.12.12.129:42269)
0429 08:00:56.818617 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.818621 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.818634 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:00:56.818755 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.818758 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.818808 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.818886 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:56.819209 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.819748 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.820353 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.820485 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.820528 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:56.820976 (+ 448us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:56.821058 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.821151 (+ 93us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.821192 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 08:00:56.826491 20823 raft_consensus.cc:1275] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Refusing update from remote peer 195cfbba0a504bc1a0b9d3c05e5c6026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:56.826512 20965 raft_consensus.cc:1275] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Refusing update from remote peer 195cfbba0a504bc1a0b9d3c05e5c6026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:56.827165 21044 consensus_queue.cc:1048] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:56.827385 21050 consensus_queue.cc:1048] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:56.833204 21053 mvcc.cc:204] Tried to move back new op lower bound from 7280433794355093504 to 7280433794247008256. Current Snapshot: MvccSnapshot[applied={T|T < 7280433794355093504}]
I20260429 08:00:56.835314 21055 mvcc.cc:204] Tried to move back new op lower bound from 7280433794355093504 to 7280433794247008256. Current Snapshot: MvccSnapshot[applied={T|T < 7280433794355093504}]
I20260429 08:00:56.841101 20521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:56.841253 20521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:56.843492 20521 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Sending DeleteTablet for 3 replicas of tablet 2945cd938f7543168891790e5509370a
I20260429 08:00:56.844177 20945 tablet_service.cc:1558] Processing DeleteTablet for tablet 2945cd938f7543168891790e5509370a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:56 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:39940
I20260429 08:00:56.844177 20661 tablet_service.cc:1558] Processing DeleteTablet for tablet 2945cd938f7543168891790e5509370a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:56 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:45920
I20260429 08:00:56.844208 20802 tablet_service.cc:1558] Processing DeleteTablet for tablet 2945cd938f7543168891790e5509370a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:56 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49594
I20260429 08:00:56.844643 21065 tablet_replica.cc:333] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: stopping tablet replica
I20260429 08:00:56.844831 21064 tablet_replica.cc:333] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: stopping tablet replica
I20260429 08:00:56.844893 21065 raft_consensus.cc:2243] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:56.845062 21064 raft_consensus.cc:2243] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:56.845165 21065 raft_consensus.cc:2272] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:56.845319 20521 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 08:00:56.845419 21064 raft_consensus.cc:2272] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:56.845710 21063 tablet_replica.cc:333] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: stopping tablet replica
W20260429 08:00:56.845731 20521 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:56.845919 21063 raft_consensus.cc:2243] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:56.846196 21063 raft_consensus.cc:2272] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:56.846729 21065 ts_tablet_manager.cc:1916] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:56.847453 21063 ts_tablet_manager.cc:1916] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:56.849392 21065 ts_tablet_manager.cc:1929] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:56.849504 21065 log.cc:1199] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/2945cd938f7543168891790e5509370a
I20260429 08:00:56.849826 21065 ts_tablet_manager.cc:1950] T 2945cd938f7543168891790e5509370a P 7ff78689746743a7be7900a6c3cfc61c: Deleting consensus metadata
I20260429 08:00:56.850054 21064 ts_tablet_manager.cc:1916] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:56.851101 20509 catalog_manager.cc:5002] TS 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611): tablet 2945cd938f7543168891790e5509370a (table test-table [id=4f68e1785bb8427f95bb4b6c18888541]) successfully deleted
I20260429 08:00:56.851686 20661 tablet_service.cc:1511] Processing CreateTablet for tablet 5c2dc84b50b24359bec5e63b44e0c5b8 (DEFAULT_TABLE table=test-table [id=43f5b5bdbf2e4c15bb7eef254adc8a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:56.851686 20945 tablet_service.cc:1511] Processing CreateTablet for tablet 5c2dc84b50b24359bec5e63b44e0c5b8 (DEFAULT_TABLE table=test-table [id=43f5b5bdbf2e4c15bb7eef254adc8a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:56.851733 20802 tablet_service.cc:1511] Processing CreateTablet for tablet 5c2dc84b50b24359bec5e63b44e0c5b8 (DEFAULT_TABLE table=test-table [id=43f5b5bdbf2e4c15bb7eef254adc8a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:56.851928 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2dc84b50b24359bec5e63b44e0c5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.851949 20802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2dc84b50b24359bec5e63b44e0c5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.851953 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2dc84b50b24359bec5e63b44e0c5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:56.853879 21043 tablet_bootstrap.cc:492] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Bootstrap starting.
I20260429 08:00:56.853888 21039 tablet_bootstrap.cc:492] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Bootstrap starting.
I20260429 08:00:56.853886 21040 tablet_bootstrap.cc:492] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Bootstrap starting.
I20260429 08:00:56.854868 21039 tablet_bootstrap.cc:654] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.854868 21040 tablet_bootstrap.cc:654] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.855062 21043 tablet_bootstrap.cc:654] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:56.855531 21063 ts_tablet_manager.cc:1929] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:56.855613 21063 log.cc:1199] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/2945cd938f7543168891790e5509370a
I20260429 08:00:56.855881 21063 ts_tablet_manager.cc:1950] T 2945cd938f7543168891790e5509370a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting consensus metadata
I20260429 08:00:56.856139 21039 tablet_bootstrap.cc:492] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: No bootstrap required, opened a new log
I20260429 08:00:56.856217 21039 ts_tablet_manager.cc:1403] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:56.856678 21039 raft_consensus.cc:359] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.856789 21039 raft_consensus.cc:385] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.856851 21039 raft_consensus.cc:740] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.856925 20509 catalog_manager.cc:5002] TS b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441): tablet 2945cd938f7543168891790e5509370a (table test-table [id=4f68e1785bb8427f95bb4b6c18888541]) successfully deleted
I20260429 08:00:56.856977 21039 consensus_queue.cc:260] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.857241 21039 ts_tablet_manager.cc:1434] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:56.857689 21040 tablet_bootstrap.cc:492] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: No bootstrap required, opened a new log
I20260429 08:00:56.857765 21040 ts_tablet_manager.cc:1403] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260429 08:00:56.857862 21064 ts_tablet_manager.cc:1929] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:56.857926 21064 log.cc:1199] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/2945cd938f7543168891790e5509370a
I20260429 08:00:56.858250 21064 ts_tablet_manager.cc:1950] T 2945cd938f7543168891790e5509370a P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting consensus metadata
I20260429 08:00:56.858211 21040 raft_consensus.cc:359] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.858306 21040 raft_consensus.cc:385] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.858336 21040 raft_consensus.cc:740] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b0969a9fd04f3aa4e5a27e0d21e31b, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.858472 21040 consensus_queue.cc:260] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.858734 21040 ts_tablet_manager.cc:1434] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:56.859150 20506 catalog_manager.cc:5002] TS 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269): tablet 2945cd938f7543168891790e5509370a (table test-table [id=4f68e1785bb8427f95bb4b6c18888541]) successfully deleted
I20260429 08:00:56.860010 21043 tablet_bootstrap.cc:492] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: No bootstrap required, opened a new log
I20260429 08:00:56.860112 21043 ts_tablet_manager.cc:1403] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260429 08:00:56.860515 21043 raft_consensus.cc:359] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.860628 21043 raft_consensus.cc:385] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:56.860690 21043 raft_consensus.cc:740] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Initialized, Role: FOLLOWER
I20260429 08:00:56.860882 21043 consensus_queue.cc:260] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.861153 21043 ts_tablet_manager.cc:1434] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:56.942873 21047 raft_consensus.cc:493] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:56.943032 21047 raft_consensus.cc:515] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.944140 21047 leader_election.cc:290] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:56.947535 20762 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.944321 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:42269 (local address 127.12.12.130:33737)
0429 08:00:56.944532 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.944547 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.944643 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.945079 (+ 436us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.945081 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.945096 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.945354 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.945361 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.946305 (+ 944us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.946308 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.947149 (+ 841us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.947158 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.947258 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.947267 (+ 9us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.947329 (+ 62us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.947384 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260429 08:00:56.947909 21069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.944468 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:35441 (local address 127.12.12.130:54329)
0429 08:00:56.944759 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:56.944771 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:56.944870 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:56.945221 (+ 351us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:56.945225 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:56.945239 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:00:56.945483 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.945494 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.946202 (+ 708us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.946207 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:56.947113 (+ 906us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:56.947122 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.947244 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.947264 (+ 20us) client_negotiation.cc:770] Sending connection context
0429 08:00:56.947329 (+ 65us) client_negotiation.cc:241] Negotiation successful
0429 08:00:56.947384 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260429 08:00:56.947997 21034 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.944837 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:54329 (local address 127.12.12.131:35441)
0429 08:00:56.944978 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.944982 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.944997 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:00:56.945052 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.945055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.945106 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.945195 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.945605 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.946069 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.947245 (+ 1176us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.947691 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.947743 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.947810 (+ 67us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.947877 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:00:56.948261 21037 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:56.944402 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:33737 (local address 127.12.12.129:42269)
0429 08:00:56.944534 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:56.944539 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:56.944727 (+ 188us) server_negotiation.cc:408] Connection header received
0429 08:00:56.944785 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:56.944791 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:56.944870 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:56.944963 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:00:56.945499 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.946200 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:56.947461 (+ 1261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:56.947978 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:56.948016 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:56.948075 (+ 59us) server_negotiation.cc:300] Negotiation successful
0429 08:00:56.948151 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:00:56.948583 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c2dc84b50b24359bec5e63b44e0c5b8" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" is_pre_election: true
I20260429 08:00:56.948753 20965 raft_consensus.cc:2468] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 0.
I20260429 08:00:56.948825 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c2dc84b50b24359bec5e63b44e0c5b8" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" is_pre_election: true
I20260429 08:00:56.949055 20681 raft_consensus.cc:2468] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 0.
I20260429 08:00:56.949172 20757 leader_election.cc:304] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff78689746743a7be7900a6c3cfc61c, b9b0969a9fd04f3aa4e5a27e0d21e31b; no voters:
I20260429 08:00:56.949419 21047 raft_consensus.cc:2804] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:56.949477 21047 raft_consensus.cc:493] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:56.949515 21047 raft_consensus.cc:3060] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.950678 21047 raft_consensus.cc:515] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.951095 21047 leader_election.cc:290] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 election: Requested vote from peers 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:56.951421 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c2dc84b50b24359bec5e63b44e0c5b8" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026"
I20260429 08:00:56.951453 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c2dc84b50b24359bec5e63b44e0c5b8" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
I20260429 08:00:56.951542 20681 raft_consensus.cc:3060] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.951558 20965 raft_consensus.cc:3060] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:56.952337 20681 raft_consensus.cc:2468] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 1.
I20260429 08:00:56.952564 20965 raft_consensus.cc:2468] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 1.
I20260429 08:00:56.952654 20754 leader_election.cc:304] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195cfbba0a504bc1a0b9d3c05e5c6026, 7ff78689746743a7be7900a6c3cfc61c; no voters:
I20260429 08:00:56.952895 21047 raft_consensus.cc:2804] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Leader election won for term 1
W20260429 08:00:56.952975 21012 tablet.cc:2404] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:56.953079 21047 raft_consensus.cc:697] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 1 LEADER]: Becoming Leader. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Running, Role: LEADER
I20260429 08:00:56.953303 21047 consensus_queue.cc:237] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:56.955569 20522 catalog_manager.cc:5671] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130). New cstate: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } health_report { overall_health: UNKNOWN } } }
W20260429 08:00:56.993866 20870 tablet.cc:2404] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449655, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.190@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:57.015777 21077 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.006859 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34411 (local address 127.0.0.1:34066)
0429 08:00:57.007102 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:57.007113 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:57.007261 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:57.007522 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:57.007525 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:57.007964 (+ 439us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:00:57.008143 (+ 179us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:57.008149 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.009006 (+ 857us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.009010 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:57.009561 (+ 551us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:57.009570 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.009956 (+ 386us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.010615 (+ 659us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:00:57.010633 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:00:57.012298 (+ 1665us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:00:57.014215 (+ 1917us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:57.014223 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:57.014228 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:57.014511 (+ 283us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:57.014777 (+ 266us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:00:57.014780 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:00:57.014783 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:00:57.014838 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:00:57.015191 (+ 353us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:00:57.015197 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:00:57.015303 (+ 106us) client_negotiation.cc:770] Sending connection context
0429 08:00:57.015416 (+ 113us) client_negotiation.cc:241] Negotiation successful
0429 08:00:57.015555 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260429 08:00:57.015825 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.006895 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.12.12.190:34411)
0429 08:00:57.007030 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.007034 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.007202 (+ 168us) server_negotiation.cc:408] Connection header received
0429 08:00:57.007362 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.007366 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.007419 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.007505 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:57.008263 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.008902 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.009693 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.009857 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.012413 (+ 2556us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:57.012432 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:57.012434 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:57.012455 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:57.014089 (+ 1634us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:57.014620 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:57.014623 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:57.014626 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:57.014679 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:57.014925 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:57.014928 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:57.014930 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:57.015088 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:57.015182 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.015482 (+ 300us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.015650 (+ 168us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":46}
W20260429 08:00:57.020489 20728 tablet.cc:2404] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:00:57.023489 20522 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:34066:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 08:00:57.024266 20522 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:57.029218 20661 tablet_service.cc:1511] Processing CreateTablet for tablet b0a258e553e848cd9dc112908da3b1ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d682be0faef4f2e9b2d24f9b6688089]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 08:00:57.029436 20802 tablet_service.cc:1511] Processing CreateTablet for tablet b0a258e553e848cd9dc112908da3b1ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d682be0faef4f2e9b2d24f9b6688089]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 08:00:57.029603 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a258e553e848cd9dc112908da3b1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.029742 20802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a258e553e848cd9dc112908da3b1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.029724 20945 tablet_service.cc:1511] Processing CreateTablet for tablet b0a258e553e848cd9dc112908da3b1ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d682be0faef4f2e9b2d24f9b6688089]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 08:00:57.029999 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a258e553e848cd9dc112908da3b1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.032047 21040 tablet_bootstrap.cc:492] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: Bootstrap starting.
I20260429 08:00:57.032204 21043 tablet_bootstrap.cc:492] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: Bootstrap starting.
I20260429 08:00:57.032334 21039 tablet_bootstrap.cc:492] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: Bootstrap starting.
I20260429 08:00:57.032991 21040 tablet_bootstrap.cc:654] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.033221 21039 tablet_bootstrap.cc:654] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.033221 21043 tablet_bootstrap.cc:654] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.034372 21043 tablet_bootstrap.cc:492] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: No bootstrap required, opened a new log
I20260429 08:00:57.034381 21040 tablet_bootstrap.cc:492] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: No bootstrap required, opened a new log
I20260429 08:00:57.034480 21043 ts_tablet_manager.cc:1403] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:57.034498 21040 ts_tablet_manager.cc:1403] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260429 08:00:57.034852 21039 tablet_bootstrap.cc:492] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: No bootstrap required, opened a new log
I20260429 08:00:57.034930 21039 ts_tablet_manager.cc:1403] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260429 08:00:57.034956 21040 raft_consensus.cc:359] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.034983 21043 raft_consensus.cc:359] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.035070 21040 raft_consensus.cc:385] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.035115 21043 raft_consensus.cc:385] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.035125 21040 raft_consensus.cc:740] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b0969a9fd04f3aa4e5a27e0d21e31b, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.035149 21043 raft_consensus.cc:740] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.035255 21040 consensus_queue.cc:260] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.035264 21043 consensus_queue.cc:260] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.035342 21039 raft_consensus.cc:359] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.035439 21039 raft_consensus.cc:385] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.035475 21039 raft_consensus.cc:740] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.035540 21040 ts_tablet_manager.cc:1434] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 08:00:57.035502 21045 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.035575 21043 ts_tablet_manager.cc:1434] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260429 08:00:57.035606 21045 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.035588 21039 consensus_queue.cc:260] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.035796 21047 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.035816 21058 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.035902 21047 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.035909 21058 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.035826 21039 ts_tablet_manager.cc:1434] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:57.199290 21047 raft_consensus.cc:493] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:57.199426 21047 raft_consensus.cc:515] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.199836 21047 leader_election.cc:290] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:57.200316 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a258e553e848cd9dc112908da3b1ff" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" is_pre_election: true
I20260429 08:00:57.200318 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a258e553e848cd9dc112908da3b1ff" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" is_pre_election: true
I20260429 08:00:57.200491 20681 raft_consensus.cc:2468] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 0.
I20260429 08:00:57.200505 20965 raft_consensus.cc:2468] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 0.
I20260429 08:00:57.200850 20754 leader_election.cc:304] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195cfbba0a504bc1a0b9d3c05e5c6026, 7ff78689746743a7be7900a6c3cfc61c; no voters:
I20260429 08:00:57.201035 21047 raft_consensus.cc:2804] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:57.201092 21047 raft_consensus.cc:493] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:57.201174 21047 raft_consensus.cc:3060] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.202036 21047 raft_consensus.cc:515] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.202404 21047 leader_election.cc:290] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 election: Requested vote from peers 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:57.202741 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a258e553e848cd9dc112908da3b1ff" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026"
I20260429 08:00:57.202775 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a258e553e848cd9dc112908da3b1ff" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
I20260429 08:00:57.202867 20681 raft_consensus.cc:3060] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.202891 20965 raft_consensus.cc:3060] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.203904 20681 raft_consensus.cc:2468] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 1.
I20260429 08:00:57.203917 20965 raft_consensus.cc:2468] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 1.
I20260429 08:00:57.204236 20757 leader_election.cc:304] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff78689746743a7be7900a6c3cfc61c, b9b0969a9fd04f3aa4e5a27e0d21e31b; no voters:
I20260429 08:00:57.204406 21047 raft_consensus.cc:2804] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:57.204504 21047 raft_consensus.cc:697] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [term 1 LEADER]: Becoming Leader. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Running, Role: LEADER
I20260429 08:00:57.204643 21047 consensus_queue.cc:237] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.205385 21070 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: TxnStatusTablet state changed. Reason: New leader 7ff78689746743a7be7900a6c3cfc61c. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.205492 21070 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: This TxnStatusTablet replica's current role is: LEADER
I20260429 08:00:57.205788 21082 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 08:00:57.205936 21082 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 08:00:57.206511 20521 catalog_manager.cc:5671] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130). New cstate: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:57.230123 21077 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.226654 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:33611 (local address 127.0.0.1:49596)
0429 08:00:57.226859 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:57.226873 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:57.226968 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:57.227283 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:57.227285 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:57.227292 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:57.227467 (+ 175us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:57.227472 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.228427 (+ 955us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.228430 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:57.228912 (+ 482us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:57.228919 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.229003 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.229036 (+ 33us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.229852 (+ 816us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.229866 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 08:00:57.229937 (+ 71us) client_negotiation.cc:241] Negotiation successful
0429 08:00:57.230009 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260429 08:00:57.230146 21038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.226769 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.12.12.130:33611)
0429 08:00:57.226905 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.226909 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.226926 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:00:57.227104 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.227108 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.227174 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.227273 (+ 99us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:57.227570 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.228318 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.229013 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.229221 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.229279 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.229741 (+ 462us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.229851 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.229965 (+ 114us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.230015 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 08:00:57.233165 20965 raft_consensus.cc:1275] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Refusing update from remote peer 7ff78689746743a7be7900a6c3cfc61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.233291 20681 raft_consensus.cc:1275] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Refusing update from remote peer 7ff78689746743a7be7900a6c3cfc61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.233618 21047 consensus_queue.cc:1048] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.233767 21070 consensus_queue.cc:1048] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.235284 21045 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: TxnStatusTablet state changed. Reason: New leader 7ff78689746743a7be7900a6c3cfc61c. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.235394 21045 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.236155 21058 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: TxnStatusTablet state changed. Reason: New leader 7ff78689746743a7be7900a6c3cfc61c. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.236260 21058 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.238188 21070 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.238298 21070 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: This TxnStatusTablet replica's current role is: LEADER
I20260429 08:00:57.238435 21070 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.238500 21070 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 7ff78689746743a7be7900a6c3cfc61c: This TxnStatusTablet replica's current role is: LEADER
I20260429 08:00:57.239231 21045 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.239328 21045 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P b9b0969a9fd04f3aa4e5a27e0d21e31b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.240552 21058 tablet_replica.cc:442] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } } }
I20260429 08:00:57.240644 21058 tablet_replica.cc:445] T b0a258e553e848cd9dc112908da3b1ff P 195cfbba0a504bc1a0b9d3c05e5c6026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 08:00:57.246394 21038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.243685 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.12.12.130:33611)
0429 08:00:57.243799 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.243803 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.243817 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:00:57.243936 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.243939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.243987 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.244069 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:57.244347 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.245121 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.245737 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.245919 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.245959 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.246032 (+ 73us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.246102 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.246235 (+ 133us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.246274 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 08:00:57.256152 21090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.253229 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:33611 (local address 127.12.12.130:41615)
0429 08:00:57.253594 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:57.253624 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:00:57.253714 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:00:57.253947 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:00:57.253949 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:00:57.253965 (+ 16us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 08:00:57.254151 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:57.254154 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.255045 (+ 891us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.255048 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:00:57.255435 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:00:57.255440 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.255535 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.255562 (+ 27us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.255903 (+ 341us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.255907 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 08:00:57.255957 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 08:00:57.255999 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260429 08:00:57.256158 21038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.253231 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41615 (local address 127.12.12.130:33611)
0429 08:00:57.253342 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.253345 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.253698 (+ 353us) server_negotiation.cc:408] Connection header received
0429 08:00:57.253792 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.253796 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.253844 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.253934 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:57.254239 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.254909 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.255529 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.255707 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.255747 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.255816 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.255892 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.255987 (+ 95us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.256023 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 08:00:57.262557 20965 raft_consensus.cc:1275] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Refusing update from remote peer 7ff78689746743a7be7900a6c3cfc61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.262674 20681 raft_consensus.cc:1275] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Refusing update from remote peer 7ff78689746743a7be7900a6c3cfc61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.263062 21087 consensus_queue.cc:1048] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.263389 21070 consensus_queue.cc:1048] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.267714 21054 mvcc.cc:204] Tried to move back new op lower bound from 7280433796144152576 to 7280433794883174400. Current Snapshot: MvccSnapshot[applied={T|T < 7280433796144152576}]
I20260429 08:00:57.309999 20522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:57.310141 20522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:57.312182 20522 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Sending DeleteTablet for 3 replicas of tablet 5c2dc84b50b24359bec5e63b44e0c5b8
I20260429 08:00:57.312762 20661 tablet_service.cc:1558] Processing DeleteTablet for tablet 5c2dc84b50b24359bec5e63b44e0c5b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:45920
I20260429 08:00:57.312848 20803 tablet_service.cc:1558] Processing DeleteTablet for tablet 5c2dc84b50b24359bec5e63b44e0c5b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49594
I20260429 08:00:57.312909 20945 tablet_service.cc:1558] Processing DeleteTablet for tablet 5c2dc84b50b24359bec5e63b44e0c5b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:39940
I20260429 08:00:57.312983 21064 tablet_replica.cc:333] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: stopping tablet replica
I20260429 08:00:57.313098 21064 raft_consensus.cc:2243] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:57.313210 21063 tablet_replica.cc:333] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: stopping tablet replica
I20260429 08:00:57.313225 21064 raft_consensus.cc:2272] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.313315 21063 raft_consensus.cc:2243] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:57.313416 21063 raft_consensus.cc:2272] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.313573 21065 tablet_replica.cc:333] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: stopping tablet replica
I20260429 08:00:57.313681 21065 raft_consensus.cc:2243] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:57.313978 21065 raft_consensus.cc:2272] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.314257 21064 ts_tablet_manager.cc:1916] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.314723 21065 ts_tablet_manager.cc:1916] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.317209 21065 ts_tablet_manager.cc:1929] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:57.317286 21065 log.cc:1199] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/5c2dc84b50b24359bec5e63b44e0c5b8
I20260429 08:00:57.317499 21064 ts_tablet_manager.cc:1929] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:57.317539 21065 ts_tablet_manager.cc:1950] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 7ff78689746743a7be7900a6c3cfc61c: Deleting consensus metadata
I20260429 08:00:57.317570 21064 log.cc:1199] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/5c2dc84b50b24359bec5e63b44e0c5b8
I20260429 08:00:57.317814 21064 ts_tablet_manager.cc:1950] T 5c2dc84b50b24359bec5e63b44e0c5b8 P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting consensus metadata
I20260429 08:00:57.317960 21063 ts_tablet_manager.cc:1916] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.318243 20509 catalog_manager.cc:5002] TS 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611): tablet 5c2dc84b50b24359bec5e63b44e0c5b8 (table test-table [id=43f5b5bdbf2e4c15bb7eef254adc8a1f]) successfully deleted
I20260429 08:00:57.318881 20506 catalog_manager.cc:5002] TS 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269): tablet 5c2dc84b50b24359bec5e63b44e0c5b8 (table test-table [id=43f5b5bdbf2e4c15bb7eef254adc8a1f]) successfully deleted
I20260429 08:00:57.320739 21063 ts_tablet_manager.cc:1929] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:57.320829 21063 log.cc:1199] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/5c2dc84b50b24359bec5e63b44e0c5b8
I20260429 08:00:57.321107 21063 ts_tablet_manager.cc:1950] T 5c2dc84b50b24359bec5e63b44e0c5b8 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting consensus metadata
I20260429 08:00:57.321862 20509 catalog_manager.cc:5002] TS b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441): tablet 5c2dc84b50b24359bec5e63b44e0c5b8 (table test-table [id=43f5b5bdbf2e4c15bb7eef254adc8a1f]) successfully deleted
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[20474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260429 08:00:57.341307 20571 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:00:57.335439 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.12.12.190:34411)
0429 08:00:57.335595 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.335598 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.335878 (+ 280us) server_negotiation.cc:408] Connection header received
0429 08:00:57.336059 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.336061 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.336112 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.336209 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:57.337057 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.337594 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.338250 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.338478 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.341173 (+ 2695us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34068: BlockingRecv error: recv got EOF from 127.0.0.1:34068 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[20474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM
I20260429 08:00:57.350461 20571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.342907 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.12.12.190:34411)
0429 08:00:57.343020 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.343023 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.343317 (+ 294us) server_negotiation.cc:408] Connection header received
0429 08:00:57.343454 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.343457 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.343509 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.343595 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:00:57.344239 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.344723 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.345378 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.345594 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.347467 (+ 1873us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:00:57.347483 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:00:57.347485 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:00:57.347508 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:00:57.348905 (+ 1397us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:57.349347 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:57.349350 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:57.349352 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:57.349404 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:00:57.349654 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:00:57.349658 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:00:57.349659 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:00:57.349801 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:00:57.349966 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.350184 (+ 218us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.350294 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 08:00:57.352860 20522 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:57.353315 20522 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:57.358436 20661 tablet_service.cc:1511] Processing CreateTablet for tablet b774e16512e14d65864c8370571bf903 (DEFAULT_TABLE table=test-table [id=58b882cd7265466fbf016b5ebeb82dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:57.358632 20803 tablet_service.cc:1511] Processing CreateTablet for tablet b774e16512e14d65864c8370571bf903 (DEFAULT_TABLE table=test-table [id=58b882cd7265466fbf016b5ebeb82dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:57.358783 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b774e16512e14d65864c8370571bf903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.358937 20803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b774e16512e14d65864c8370571bf903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.360590 21043 tablet_bootstrap.cc:492] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Bootstrap starting.
I20260429 08:00:57.361277 20945 tablet_service.cc:1511] Processing CreateTablet for tablet b774e16512e14d65864c8370571bf903 (DEFAULT_TABLE table=test-table [id=58b882cd7265466fbf016b5ebeb82dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:57.361478 21043 tablet_bootstrap.cc:654] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.361537 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b774e16512e14d65864c8370571bf903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.362543 21043 tablet_bootstrap.cc:492] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: No bootstrap required, opened a new log
I20260429 08:00:57.362646 21043 ts_tablet_manager.cc:1403] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:57.363166 21043 raft_consensus.cc:359] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.363287 21043 raft_consensus.cc:385] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.363328 21043 raft_consensus.cc:740] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.363454 21043 consensus_queue.cc:260] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.363705 21043 ts_tablet_manager.cc:1434] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:57.363801 21040 tablet_bootstrap.cc:492] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Bootstrap starting.
I20260429 08:00:57.364742 21040 tablet_bootstrap.cc:654] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.365834 21039 tablet_bootstrap.cc:492] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Bootstrap starting.
I20260429 08:00:57.366761 21039 tablet_bootstrap.cc:654] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.367597 21040 tablet_bootstrap.cc:492] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: No bootstrap required, opened a new log
I20260429 08:00:57.367700 21040 ts_tablet_manager.cc:1403] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260429 08:00:57.368172 21040 raft_consensus.cc:359] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.368290 21040 raft_consensus.cc:385] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.368323 21040 raft_consensus.cc:740] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b0969a9fd04f3aa4e5a27e0d21e31b, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.368462 21040 consensus_queue.cc:260] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.368724 21040 ts_tablet_manager.cc:1434] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:57.369388 21039 tablet_bootstrap.cc:492] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: No bootstrap required, opened a new log
I20260429 08:00:57.369493 21039 ts_tablet_manager.cc:1403] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260429 08:00:57.369903 21039 raft_consensus.cc:359] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.370014 21039 raft_consensus.cc:385] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.370060 21039 raft_consensus.cc:740] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.370196 21039 consensus_queue.cc:260] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.370436 21039 ts_tablet_manager.cc:1434] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:57.374387 21058 raft_consensus.cc:493] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:57.374490 21058 raft_consensus.cc:515] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.374929 21058 leader_election.cc:290] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:57.375329 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b774e16512e14d65864c8370571bf903" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" is_pre_election: true
I20260429 08:00:57.375344 20823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b774e16512e14d65864c8370571bf903" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff78689746743a7be7900a6c3cfc61c" is_pre_election: true
I20260429 08:00:57.375452 20965 raft_consensus.cc:2468] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 0.
I20260429 08:00:57.375468 20823 raft_consensus.cc:2468] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 0.
I20260429 08:00:57.375762 20615 leader_election.cc:304] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195cfbba0a504bc1a0b9d3c05e5c6026, 7ff78689746743a7be7900a6c3cfc61c; no voters:
I20260429 08:00:57.375967 21058 raft_consensus.cc:2804] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:57.376036 21058 raft_consensus.cc:493] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:57.376084 21058 raft_consensus.cc:3060] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.377107 21058 raft_consensus.cc:515] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.377431 21058 leader_election.cc:290] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 election: Requested vote from peers 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:57.377839 20823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b774e16512e14d65864c8370571bf903" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff78689746743a7be7900a6c3cfc61c"
I20260429 08:00:57.377841 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b774e16512e14d65864c8370571bf903" candidate_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
I20260429 08:00:57.377957 20965 raft_consensus.cc:3060] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.377961 20823 raft_consensus.cc:3060] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.378820 20823 raft_consensus.cc:2468] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 1.
I20260429 08:00:57.379012 20965 raft_consensus.cc:2468] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195cfbba0a504bc1a0b9d3c05e5c6026 in term 1.
I20260429 08:00:57.379283 20615 leader_election.cc:304] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195cfbba0a504bc1a0b9d3c05e5c6026, 7ff78689746743a7be7900a6c3cfc61c; no voters:
I20260429 08:00:57.379525 21058 raft_consensus.cc:2804] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:57.379743 21058 raft_consensus.cc:697] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 LEADER]: Becoming Leader. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Running, Role: LEADER
I20260429 08:00:57.379886 21058 consensus_queue.cc:237] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.381330 20521 catalog_manager.cc:5671] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 reported cstate change: term changed from 0 to 1, leader changed from <none> to 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129). New cstate: current_term: 1 leader_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:57.393195 21037 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.390285 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.12.12.129:42269)
0429 08:00:57.390420 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.390425 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.390442 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:00:57.390543 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.390549 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.390604 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.390694 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:57.391065 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.391798 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.392427 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.392617 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.392681 (+ 64us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.392762 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.392895 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.393008 (+ 113us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.393066 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 08:00:57.395236 20823 raft_consensus.cc:1275] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Refusing update from remote peer 195cfbba0a504bc1a0b9d3c05e5c6026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.395298 20965 raft_consensus.cc:1275] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Refusing update from remote peer 195cfbba0a504bc1a0b9d3c05e5c6026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.395702 21058 consensus_queue.cc:1048] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.395838 21110 consensus_queue.cc:1048] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.403288 20520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:57.403462 20520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:57.406000 20520 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Sending DeleteTablet for 3 replicas of tablet b774e16512e14d65864c8370571bf903
I20260429 08:00:57.406697 20661 tablet_service.cc:1558] Processing DeleteTablet for tablet b774e16512e14d65864c8370571bf903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:45920
I20260429 08:00:57.406898 20945 tablet_service.cc:1558] Processing DeleteTablet for tablet b774e16512e14d65864c8370571bf903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:39940
I20260429 08:00:57.406956 21064 tablet_replica.cc:333] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: stopping tablet replica
I20260429 08:00:57.407066 21063 tablet_replica.cc:333] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: stopping tablet replica
I20260429 08:00:57.407079 21064 raft_consensus.cc:2243] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:57.407155 21063 raft_consensus.cc:2243] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:57.407251 21063 raft_consensus.cc:2272] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.407248 21064 raft_consensus.cc:2272] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.407909 20803 tablet_service.cc:1558] Processing DeleteTablet for tablet b774e16512e14d65864c8370571bf903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49594
I20260429 08:00:57.408062 21063 ts_tablet_manager.cc:1916] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.408073 20802 tablet_service.cc:1558] Processing DeleteTablet for tablet b774e16512e14d65864c8370571bf903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49594
I20260429 08:00:57.408208 21065 tablet_replica.cc:333] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: stopping tablet replica
I20260429 08:00:57.408318 21065 raft_consensus.cc:2243] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:57.408448 21065 raft_consensus.cc:2272] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.409204 21065 ts_tablet_manager.cc:1916] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.410187 21063 ts_tablet_manager.cc:1929] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:57.410246 21063 log.cc:1199] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/b774e16512e14d65864c8370571bf903
I20260429 08:00:57.410485 21063 ts_tablet_manager.cc:1950] T b774e16512e14d65864c8370571bf903 P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting consensus metadata
I20260429 08:00:57.411175 20509 catalog_manager.cc:5002] TS b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441): tablet b774e16512e14d65864c8370571bf903 (table test-table [id=58b882cd7265466fbf016b5ebeb82dc0]) successfully deleted
I20260429 08:00:57.411342 21065 ts_tablet_manager.cc:1929] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:57.411388 21065 log.cc:1199] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/b774e16512e14d65864c8370571bf903
I20260429 08:00:57.411566 21065 ts_tablet_manager.cc:1950] T b774e16512e14d65864c8370571bf903 P 7ff78689746743a7be7900a6c3cfc61c: Deleting consensus metadata
I20260429 08:00:57.412115 20509 catalog_manager.cc:5002] TS 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611): tablet b774e16512e14d65864c8370571bf903 (table test-table [id=58b882cd7265466fbf016b5ebeb82dc0]) successfully deleted
W20260429 08:00:57.412374 20509 catalog_manager.cc:4969] TS 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611): delete failed for tablet b774e16512e14d65864c8370571bf903 because the tablet was not found. No further retry: Not found: Tablet not found: b774e16512e14d65864c8370571bf903
I20260429 08:00:57.412565 21064 ts_tablet_manager.cc:1916] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.413930 20520 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:00:57.414372 20520 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:00:57.414676 21064 ts_tablet_manager.cc:1929] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:00:57.414772 21064 log.cc:1199] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-0/wal/wals/b774e16512e14d65864c8370571bf903
I20260429 08:00:57.415046 21064 ts_tablet_manager.cc:1950] T b774e16512e14d65864c8370571bf903 P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting consensus metadata
I20260429 08:00:57.415674 20506 catalog_manager.cc:5002] TS 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269): tablet b774e16512e14d65864c8370571bf903 (table test-table [id=58b882cd7265466fbf016b5ebeb82dc0]) successfully deleted
I20260429 08:00:57.419363 20661 tablet_service.cc:1511] Processing CreateTablet for tablet 50c08145273d491d840e229b5f5a560a (DEFAULT_TABLE table=test-table [id=0fa863e297fe470f89e2c3435dfb41f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:57.419620 20661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c08145273d491d840e229b5f5a560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.419628 20802 tablet_service.cc:1511] Processing CreateTablet for tablet 50c08145273d491d840e229b5f5a560a (DEFAULT_TABLE table=test-table [id=0fa863e297fe470f89e2c3435dfb41f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:57.419867 20802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c08145273d491d840e229b5f5a560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.419796 20945 tablet_service.cc:1511] Processing CreateTablet for tablet 50c08145273d491d840e229b5f5a560a (DEFAULT_TABLE table=test-table [id=0fa863e297fe470f89e2c3435dfb41f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:00:57.420056 20945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c08145273d491d840e229b5f5a560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:00:57.422247 21043 tablet_bootstrap.cc:492] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Bootstrap starting.
I20260429 08:00:57.422842 21040 tablet_bootstrap.cc:492] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Bootstrap starting.
I20260429 08:00:57.423159 21043 tablet_bootstrap.cc:654] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.423687 21039 tablet_bootstrap.cc:492] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: Bootstrap starting.
I20260429 08:00:57.423945 21040 tablet_bootstrap.cc:654] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.424517 21043 tablet_bootstrap.cc:492] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: No bootstrap required, opened a new log
I20260429 08:00:57.424569 21039 tablet_bootstrap.cc:654] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: Neither blocks nor log segments found. Creating new log.
I20260429 08:00:57.424623 21043 ts_tablet_manager.cc:1403] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 08:00:57.425094 21043 raft_consensus.cc:359] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.425202 21043 raft_consensus.cc:385] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.425231 21043 raft_consensus.cc:740] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.425326 21040 tablet_bootstrap.cc:492] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: No bootstrap required, opened a new log
I20260429 08:00:57.425395 21040 ts_tablet_manager.cc:1403] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260429 08:00:57.425352 21043 consensus_queue.cc:260] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.425611 21043 ts_tablet_manager.cc:1434] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 08:00:57.425796 21040 raft_consensus.cc:359] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.425885 21040 raft_consensus.cc:385] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.425912 21040 raft_consensus.cc:740] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b0969a9fd04f3aa4e5a27e0d21e31b, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.426016 21040 consensus_queue.cc:260] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.426232 21040 ts_tablet_manager.cc:1434] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 08:00:57.435986 21039 tablet_bootstrap.cc:492] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: No bootstrap required, opened a new log
I20260429 08:00:57.436103 21039 ts_tablet_manager.cc:1403] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260429 08:00:57.436556 21039 raft_consensus.cc:359] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.436669 21039 raft_consensus.cc:385] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:00:57.436692 21039 raft_consensus.cc:740] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195cfbba0a504bc1a0b9d3c05e5c6026, State: Initialized, Role: FOLLOWER
I20260429 08:00:57.436954 21039 consensus_queue.cc:260] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.437752 21039 ts_tablet_manager.cc:1434] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 08:00:57.491204 21087 raft_consensus.cc:493] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:00:57.491372 21087 raft_consensus.cc:515] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.491894 21087 leader_election.cc:290] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:57.492338 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c08145273d491d840e229b5f5a560a" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" is_pre_election: true
I20260429 08:00:57.492444 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c08145273d491d840e229b5f5a560a" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" is_pre_election: true
I20260429 08:00:57.492522 20681 raft_consensus.cc:2468] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 0.
I20260429 08:00:57.492604 20965 raft_consensus.cc:2468] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 0.
I20260429 08:00:57.492990 20757 leader_election.cc:304] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff78689746743a7be7900a6c3cfc61c, b9b0969a9fd04f3aa4e5a27e0d21e31b; no voters:
I20260429 08:00:57.493188 21087 raft_consensus.cc:2804] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:00:57.493235 21087 raft_consensus.cc:493] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:00:57.493273 21087 raft_consensus.cc:3060] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.494446 21087 raft_consensus.cc:515] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.494812 21087 leader_election.cc:290] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 election: Requested vote from peers 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269), b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441)
I20260429 08:00:57.495154 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c08145273d491d840e229b5f5a560a" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026"
I20260429 08:00:57.495258 20681 raft_consensus.cc:3060] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.495225 20965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c08145273d491d840e229b5f5a560a" candidate_uuid: "7ff78689746743a7be7900a6c3cfc61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b"
I20260429 08:00:57.495329 20965 raft_consensus.cc:3060] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:00:57.496279 20681 raft_consensus.cc:2468] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 1.
I20260429 08:00:57.496312 20965 raft_consensus.cc:2468] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff78689746743a7be7900a6c3cfc61c in term 1.
I20260429 08:00:57.496596 20757 leader_election.cc:304] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff78689746743a7be7900a6c3cfc61c, b9b0969a9fd04f3aa4e5a27e0d21e31b; no voters:
I20260429 08:00:57.496845 21087 raft_consensus.cc:2804] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:00:57.496927 21087 raft_consensus.cc:697] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 1 LEADER]: Becoming Leader. State: Replica: 7ff78689746743a7be7900a6c3cfc61c, State: Running, Role: LEADER
I20260429 08:00:57.497068 21087 consensus_queue.cc:237] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } }
I20260429 08:00:57.498636 20520 catalog_manager.cc:5671] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130). New cstate: current_term: 1 leader_uuid: "7ff78689746743a7be7900a6c3cfc61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff78689746743a7be7900a6c3cfc61c" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 33611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 } health_report { overall_health: UNKNOWN } } }
I20260429 08:00:57.525521 21038 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.522927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.12.12.130:33611)
0429 08:00:57.523068 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 08:00:57.523072 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:00:57.523085 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:00:57.523215 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:00:57.523220 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:00:57.523276 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:00:57.523374 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:00:57.523705 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.524164 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:00:57.524800 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:00:57.525015 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:00:57.525054 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:00:57.525115 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:00:57.525206 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0429 08:00:57.525328 (+ 122us) server_negotiation.cc:300] Negotiation successful
0429 08:00:57.525368 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:00:57.531495 20681 raft_consensus.cc:1275] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Refusing update from remote peer 7ff78689746743a7be7900a6c3cfc61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.531808 20965 raft_consensus.cc:1275] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Refusing update from remote peer 7ff78689746743a7be7900a6c3cfc61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:00:57.532003 21094 consensus_queue.cc:1048] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "195cfbba0a504bc1a0b9d3c05e5c6026" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.532174 21047 consensus_queue.cc:1048] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b0969a9fd04f3aa4e5a27e0d21e31b" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:00:57.564638 20521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:57.564828 20521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:00:57.566816 20521 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ab69a0bff21940ca86f1f7a034fe8132: Sending DeleteTablet for 3 replicas of tablet 50c08145273d491d840e229b5f5a560a
I20260429 08:00:57.567399 20661 tablet_service.cc:1558] Processing DeleteTablet for tablet 50c08145273d491d840e229b5f5a560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:45920
I20260429 08:00:57.567548 20945 tablet_service.cc:1558] Processing DeleteTablet for tablet 50c08145273d491d840e229b5f5a560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:39940
I20260429 08:00:57.567601 21064 tablet_replica.cc:333] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: stopping tablet replica
I20260429 08:00:57.567727 21063 tablet_replica.cc:333] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: stopping tablet replica
I20260429 08:00:57.567727 21064 raft_consensus.cc:2243] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:57.567728 20802 tablet_service.cc:1558] Processing DeleteTablet for tablet 50c08145273d491d840e229b5f5a560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:00:57 UTC) from {username='oryx', principal='oryx/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:49594
I20260429 08:00:57.567822 21063 raft_consensus.cc:2243] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:00:57.567849 21064 raft_consensus.cc:2272] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.567878 21065 tablet_replica.cc:333] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: stopping tablet replica
I20260429 08:00:57.567927 21063 raft_consensus.cc:2272] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.567976 21065 raft_consensus.cc:2243] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:00:57.568197 21065 raft_consensus.cc:2272] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:00:57.568899 21063 ts_tablet_manager.cc:1916] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.569202 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 20596
I20260429 08:00:57.569203 21064 ts_tablet_manager.cc:1916] T 50c08145273d491d840e229b5f5a560a P 195cfbba0a504bc1a0b9d3c05e5c6026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.571733 21063 ts_tablet_manager.cc:1929] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:57.571805 21063 log.cc:1199] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/50c08145273d491d840e229b5f5a560a
I20260429 08:00:57.572045 21063 ts_tablet_manager.cc:1950] T 50c08145273d491d840e229b5f5a560a P b9b0969a9fd04f3aa4e5a27e0d21e31b: Deleting consensus metadata
I20260429 08:00:57.572721 20509 catalog_manager.cc:5002] TS b9b0969a9fd04f3aa4e5a27e0d21e31b (127.12.12.131:35441): tablet 50c08145273d491d840e229b5f5a560a (table test-table [id=0fa863e297fe470f89e2c3435dfb41f6]) successfully deleted
I20260429 08:00:57.576439 21065 ts_tablet_manager.cc:1916] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:00:57.579344 21065 ts_tablet_manager.cc:1929] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 08:00:57.579427 21065 log.cc:1199] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777449587217345-12338-0/minicluster-data/ts-1/wal/wals/50c08145273d491d840e229b5f5a560a
W20260429 08:00:57.579610 20506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.129:42269 (error 108)
I20260429 08:00:57.579684 21065 ts_tablet_manager.cc:1950] T 50c08145273d491d840e229b5f5a560a P 7ff78689746743a7be7900a6c3cfc61c: Deleting consensus metadata
I20260429 08:00:57.580462 20509 catalog_manager.cc:5002] TS 7ff78689746743a7be7900a6c3cfc61c (127.12.12.130:33611): tablet 50c08145273d491d840e229b5f5a560a (table test-table [id=0fa863e297fe470f89e2c3435dfb41f6]) successfully deleted
I20260429 08:00:57.580895 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 20738
I20260429 08:00:57.580938 21123 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:00:57.580371 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:42269 (local address 127.0.0.1:45952)
0429 08:00:57.580745 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0429 08:00:57.580836 (+ 91us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":53,"threads_started":1}
W20260429 08:00:57.581032 20506 catalog_manager.cc:4729] TS 195cfbba0a504bc1a0b9d3c05e5c6026 (127.12.12.129:42269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50c08145273d491d840e229b5f5a560a: Network error: Client connection negotiation failed: client connection to 127.12.12.129:42269: connect: Connection refused (error 111)
I20260429 08:00:57.588941 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 20880
I20260429 08:00:57.595048 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 20490
2026-04-29T08:00:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3871 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[21126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[21126](info): set up 2 sockets
Apr 29 08:00:57 dist-test-slave-nhqz krb5kdc[21126](info): commencing operation
krb5kdc: starting...
W20260429 08:00:59.574261 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.956s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:00:59 dist-test-slave-nhqz krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:00:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:00:59.711300 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:33689
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:00:59.812299 21142 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:00:59.812564 21142 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:00:59.812685 21142 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:00:59.815927 21142 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:00:59.816018 21142 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:00:59.816064 21142 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:00:59.816128 21142 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:00:59.816172 21142 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:00:59.820372 21142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:00:59.821465 21142 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:00:59.822285 21142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:00:59.828380 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:59.828377 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:00:59.828380 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:00:59.828886 21142 server_base.cc:1061] running on GCE node
I20260429 08:00:59.829492 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:00:59.830621 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:00:59.831815 21142 hybrid_clock.cc:648] HybridClock initialized: now 1777449659831785 us; error 44 us; skew 500 ppm
W20260429 08:00:59.835168 21142 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-04-29T08:00:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2242 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:00:59 dist-test-slave-nhqz krb5kdc[21155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:00:59 dist-test-slave-nhqz krb5kdc[21155](info): set up 2 sockets
Apr 29 08:00:59 dist-test-slave-nhqz krb5kdc[21155](info): commencing operation
krb5kdc: starting...
W20260429 08:01:01.814275 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.954s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:01:01 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:01:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:01.955551 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34401
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:02.057150 21171 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:02.057386 21171 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:02.057492 21171 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:02.060699 21171 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:01:02.060783 21171 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:02.060824 21171 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:02.060906 21171 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:01:02.060941 21171 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:01:02.065167 21171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:02.066226 21171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:02.067056 21171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:02.072698 21177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.072688 21179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.072670 21176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:02.073065 21171 server_base.cc:1061] running on GCE node
I20260429 08:01:02.073513 21171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:02.074332 21171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:02.075521 21171 hybrid_clock.cc:648] HybridClock initialized: now 1777449662075499 us; error 36 us; skew 500 ppm
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:02.078085 21171 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:01:02.079049 21171 webserver.cc:492] Webserver started at http://127.12.12.190:33019/ using document root <none> and password file <none>
I20260429 08:01:02.079528 21171 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:02.079571 21171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:02.079722 21171 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:02.081398 21171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "52adb3700fb143edbd535e65baf28b2f"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "42932ca2f71a13b8db82eb5ece99f535"
server_key_iv: "4f1c1986738f1314b4195cf89feb3773"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.081815 21171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "52adb3700fb143edbd535e65baf28b2f"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "42932ca2f71a13b8db82eb5ece99f535"
server_key_iv: "4f1c1986738f1314b4195cf89feb3773"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.084944 21171 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:01:02.086969 21186 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.088002 21171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:02.088094 21171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "52adb3700fb143edbd535e65baf28b2f"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "42932ca2f71a13b8db82eb5ece99f535"
server_key_iv: "4f1c1986738f1314b4195cf89feb3773"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.088217 21171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:02.099439 21171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:02.101771 21171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:02.101954 21171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:02.108913 21171 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:34401
I20260429 08:01:02.108925 21238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:34401 every 8 connection(s)
I20260429 08:01:02.109855 21171 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:01:02.111235 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21171
I20260429 08:01:02.111367 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:01:02.111650 12338 external_mini_cluster.cc:1468] Setting key 68b90688dd303992f1a8c174e4b3df1f
I20260429 08:01:02.112907 21239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:02.117527 21239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Bootstrap starting.
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:01:02.119982 21239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:02.120858 21239 log.cc:826] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:02.122617 21239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: No bootstrap required, opened a new log
I20260429 08:01:02.124799 21242 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.112935 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.12.12.190:34401)
0429 08:01:02.113332 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:02.113341 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:02.113372 (+ 31us) server_negotiation.cc:408] Connection header received
0429 08:01:02.114390 (+ 1018us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:02.114408 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:02.114668 (+ 260us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:02.115012 (+ 344us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:02.115934 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.116667 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.117388 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.117729 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.120033 (+ 2304us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:02.120051 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:02.120061 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:02.120117 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:02.122054 (+ 1937us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.122487 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.122491 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.122495 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.122560 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.122848 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.122852 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.122854 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.123300 (+ 446us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:02.123504 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:02.123833 (+ 329us) server_negotiation.cc:300] Negotiation successful
0429 08:01:02.124073 (+ 240us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260429 08:01:02.125994 21239 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:02.126163 21239 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:02.126197 21239 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52adb3700fb143edbd535e65baf28b2f, State: Initialized, Role: FOLLOWER
I20260429 08:01:02.126673 21239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:02.126838 21239 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:01:02.126900 21239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:01:02.127005 21239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:02.128276 21239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:02.128619 21239 leader_election.cc:304] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52adb3700fb143edbd535e65baf28b2f; no voters:
I20260429 08:01:02.129017 21239 leader_election.cc:290] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:01:02.129158 21244 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:01:02.129405 21244 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 LEADER]: Becoming Leader. State: Replica: 52adb3700fb143edbd535e65baf28b2f, State: Running, Role: LEADER
I20260429 08:01:02.129733 21244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:02.130157 21239 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:01:02.131263 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52adb3700fb143edbd535e65baf28b2f. Latest consensus state: current_term: 1 leader_uuid: "52adb3700fb143edbd535e65baf28b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } } }
I20260429 08:01:02.131405 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:02.131721 21245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52adb3700fb143edbd535e65baf28b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } } }
I20260429 08:01:02.131804 21245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:02.133067 21253 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:01:02.135469 21253 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:01:02.140285 21253 catalog_manager.cc:1357] Generated new cluster ID: 8818857c55d84a6b837dca30f9f6c2ba
I20260429 08:01:02.140358 21253 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:01:02.152333 21253 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:01:02.153630 21253 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:01:02.168118 21253 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Generated new TSK 0
I20260429 08:01:02.168906 21253 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:02.217991 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34401
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:02.324451 21267 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:02.324673 21267 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:02.324751 21267 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:02.328048 21267 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:02.328110 21267 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:02.328188 21267 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:01:02.332515 21267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:02.333704 21267 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:02.334537 21267 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:02.340969 21272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.340975 21275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.340994 21273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:02.341383 21267 server_base.cc:1061] running on GCE node
I20260429 08:01:02.341764 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:02.342288 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:02.343470 21267 hybrid_clock.cc:648] HybridClock initialized: now 1777449662343449 us; error 33 us; skew 500 ppm
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:02.349334 21267 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:01:02.350478 21267 webserver.cc:492] Webserver started at http://127.12.12.129:45719/ using document root <none> and password file <none>
I20260429 08:01:02.351042 21267 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:02.351105 21267 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:02.351316 21267 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:02.353094 21267 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "c9bb06bb74f24505ad768c8eb15b2ad5"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "fd5a0ffe9e99f1db5db80287ccd858aa"
server_key_iv: "f3aecc8b4da731917b36eeb52749aaca"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.353632 21267 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "c9bb06bb74f24505ad768c8eb15b2ad5"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "fd5a0ffe9e99f1db5db80287ccd858aa"
server_key_iv: "f3aecc8b4da731917b36eeb52749aaca"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.356796 21267 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:01:02.359043 21282 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.359983 21267 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:02.360114 21267 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "c9bb06bb74f24505ad768c8eb15b2ad5"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "fd5a0ffe9e99f1db5db80287ccd858aa"
server_key_iv: "f3aecc8b4da731917b36eeb52749aaca"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.360214 21267 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:02.372187 21267 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:02.375140 21267 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:02.375375 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:02.375991 21267 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:02.377005 21267 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:02.377082 21267 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.377156 21267 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:02.377215 21267 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.387600 21267 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:36525
I20260429 08:01:02.387629 21395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:36525 every 8 connection(s)
I20260429 08:01:02.388721 21267 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:01:02.394482 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21267
I20260429 08:01:02.394654 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:01:02.394948 12338 external_mini_cluster.cc:1468] Setting key d77025d4b4b3dbf1779228ade6f27280
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:01:02.404430 21242 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.390581 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:50595 (local address 127.12.12.190:34401)
0429 08:01:02.390758 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:02.390763 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:02.391565 (+ 802us) server_negotiation.cc:408] Connection header received
0429 08:01:02.392857 (+ 1292us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:02.392864 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:02.392932 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:02.393042 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:02.395299 (+ 2257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.396098 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.397092 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.397596 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.400332 (+ 2736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:02.400362 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:02.400365 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:02.400391 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:02.402303 (+ 1912us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.402895 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.402899 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.402901 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.402958 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.403343 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.403348 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.403349 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.403513 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:02.403624 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:02.404119 (+ 495us) server_negotiation.cc:300] Negotiation successful
0429 08:01:02.404253 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 08:01:02.405169 21398 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.390898 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:50595)
0429 08:01:02.391376 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:02.391423 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:02.392567 (+ 1144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:02.393262 (+ 695us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:02.393278 (+ 16us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:02.393859 (+ 581us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:02.395096 (+ 1237us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:02.395114 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.396280 (+ 1166us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.396285 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:02.396915 (+ 630us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:02.396925 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.397180 (+ 255us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.397969 (+ 789us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:02.398026 (+ 57us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:02.400158 (+ 2132us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:02.402462 (+ 2304us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:02.402468 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:02.402481 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:02.402743 (+ 262us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:02.403095 (+ 352us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:02.403097 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:02.403099 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:02.403201 (+ 102us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:02.403626 (+ 425us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:02.403635 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:02.403877 (+ 242us) client_negotiation.cc:770] Sending connection context
0429 08:01:02.404089 (+ 212us) client_negotiation.cc:241] Negotiation successful
0429 08:01:02.404332 (+ 243us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:01:02.406909 21396 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34401
I20260429 08:01:02.407217 21396 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:02.407929 21396 heartbeater.cc:507] Master 127.12.12.190:34401 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
I20260429 08:01:02.409976 21203 ts_manager.cc:194] Registered new tserver with Master: c9bb06bb74f24505ad768c8eb15b2ad5 (127.12.12.129:36525)
I20260429 08:01:02.411599 21203 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:50595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:02.455492 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34401
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:02.555802 21403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:02.556020 21403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:02.556077 21403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:02.559219 21403 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:02.559278 21403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:02.559355 21403 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:01:02.563359 21403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:02.564394 21403 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:02.565191 21403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:02.571273 21409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.571286 21411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.571544 21408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:02.571663 21403 server_base.cc:1061] running on GCE node
I20260429 08:01:02.572129 21403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:02.572618 21403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:02.573783 21403 hybrid_clock.cc:648] HybridClock initialized: now 1777449662573771 us; error 26 us; skew 500 ppm
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:02.576195 21403 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:01:02.577235 21403 webserver.cc:492] Webserver started at http://127.12.12.130:37363/ using document root <none> and password file <none>
I20260429 08:01:02.577777 21403 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:02.577850 21403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:02.578047 21403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:02.579596 21403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "e45743dae31d4d51a502e599ababcc30"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "893888853971d5775fba7b1ea61d55bc"
server_key_iv: "cd8dd648bc8507bdf3e12d71e30900e0"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.580050 21403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "e45743dae31d4d51a502e599ababcc30"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "893888853971d5775fba7b1ea61d55bc"
server_key_iv: "cd8dd648bc8507bdf3e12d71e30900e0"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.583081 21403 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260429 08:01:02.585146 21418 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.586054 21403 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:02.586179 21403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "e45743dae31d4d51a502e599ababcc30"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "893888853971d5775fba7b1ea61d55bc"
server_key_iv: "cd8dd648bc8507bdf3e12d71e30900e0"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.586280 21403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:02.604780 21403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:02.607402 21403 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:02.607609 21403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:02.608172 21403 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:02.609134 21403 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:02.609203 21403 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.609269 21403 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:02.609313 21403 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.618016 21403 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:32965
I20260429 08:01:02.618041 21531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:32965 every 8 connection(s)
I20260429 08:01:02.618938 21403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:01:02.621315 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21403
I20260429 08:01:02.621438 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:01:02.621706 12338 external_mini_cluster.cc:1468] Setting key a312a2af135bff5d759051348c377f96
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:01:02.631759 21242 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.620605 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:53313 (local address 127.12.12.190:34401)
0429 08:01:02.620757 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:02.620760 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:02.621410 (+ 650us) server_negotiation.cc:408] Connection header received
0429 08:01:02.622469 (+ 1059us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:02.622473 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:02.622533 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:02.622617 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:02.624249 (+ 1632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.625037 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.625844 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.626035 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.628216 (+ 2181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:02.628241 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:02.628243 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:02.628268 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:02.629774 (+ 1506us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.630253 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.630257 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.630259 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.630308 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.630715 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.630717 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.630719 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.630902 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:02.631007 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:02.631509 (+ 502us) server_negotiation.cc:300] Negotiation successful
0429 08:01:02.631609 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 08:01:02.632601 21534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.620887 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:53313)
0429 08:01:02.621270 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:02.621310 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:02.622270 (+ 960us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:02.622786 (+ 516us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:02.622794 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:02.623267 (+ 473us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:02.624101 (+ 834us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:02.624113 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.625203 (+ 1090us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.625210 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:02.625718 (+ 508us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:02.625725 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.625951 (+ 226us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.626583 (+ 632us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:02.626602 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:02.628057 (+ 1455us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:02.629911 (+ 1854us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:02.629916 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:02.629928 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:02.630150 (+ 222us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:02.630436 (+ 286us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:02.630440 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:02.630443 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:02.630582 (+ 139us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:02.631007 (+ 425us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:02.631012 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:02.631227 (+ 215us) client_negotiation.cc:770] Sending connection context
0429 08:01:02.631436 (+ 209us) client_negotiation.cc:241] Negotiation successful
0429 08:01:02.631624 (+ 188us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:01:02.633948 21532 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34401
I20260429 08:01:02.634192 21532 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:02.634680 21532 heartbeater.cc:507] Master 127.12.12.190:34401 requested a full tablet report, sending...
I20260429 08:01:02.635764 21203 ts_manager.cc:194] Registered new tserver with Master: e45743dae31d4d51a502e599ababcc30 (127.12.12.130:32965)
I20260429 08:01:02.636302 21203 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:53313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:02.669804 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34401
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:02.773733 21539 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:02.773926 21539 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:02.773980 21539 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:02.777200 21539 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:02.777259 21539 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:02.777338 21539 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:01:02.781459 21539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:02.782438 21539 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:02.783288 21539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:02.789976 21544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.789944 21547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:02.789944 21545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:02.790246 21539 server_base.cc:1061] running on GCE node
I20260429 08:01:02.790591 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:02.791144 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:02.792328 21539 hybrid_clock.cc:648] HybridClock initialized: now 1777449662792305 us; error 39 us; skew 500 ppm
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:02.795128 21539 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:01:02.796214 21539 webserver.cc:492] Webserver started at http://127.12.12.131:38161/ using document root <none> and password file <none>
I20260429 08:01:02.796797 21539 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:02.796918 21539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:02.797114 21539 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:02.798820 21539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "e86984f8e3394023a91521fa71743ebb"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "dd785796d22a51c1f2a7987f2480913c"
server_key_iv: "ca0af4d3ddaaf82d94de6eaec9566430"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.799283 21539 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "e86984f8e3394023a91521fa71743ebb"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "dd785796d22a51c1f2a7987f2480913c"
server_key_iv: "ca0af4d3ddaaf82d94de6eaec9566430"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.802695 21539 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 08:01:02.804927 21554 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.805958 21539 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:02.806097 21539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "e86984f8e3394023a91521fa71743ebb"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "dd785796d22a51c1f2a7987f2480913c"
server_key_iv: "ca0af4d3ddaaf82d94de6eaec9566430"
server_key_version: "encryptionkey@0"
I20260429 08:01:02.806206 21539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:02.827818 21539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:02.835069 21539 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:02.835304 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:02.835929 21539 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:02.836975 21539 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:02.837044 21539 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.837112 21539 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:02.837164 21539 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:02.846321 21539 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:34669
I20260429 08:01:02.846344 21667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:34669 every 8 connection(s)
I20260429 08:01:02.847297 21539 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
Apr 29 08:01:02 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:01:02.856088 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21539
I20260429 08:01:02.856241 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:01:02.856621 12338 external_mini_cluster.cc:1468] Setting key f7527dbcf8007bebd88db2550eaabb16
I20260429 08:01:02.859802 21242 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.849116 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:35235 (local address 127.12.12.190:34401)
0429 08:01:02.849266 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:02.849269 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:02.849930 (+ 661us) server_negotiation.cc:408] Connection header received
0429 08:01:02.850798 (+ 868us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:02.850801 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:02.850865 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:02.850960 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:02.852501 (+ 1541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.853048 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.853710 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.853894 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.856045 (+ 2151us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:02.856070 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:02.856073 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:02.856097 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:02.857935 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.858488 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.858492 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.858494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.858543 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:02.858881 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:02.858884 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:02.858885 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:02.859057 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:02.859150 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:02.859595 (+ 445us) server_negotiation.cc:300] Negotiation successful
0429 08:01:02.859685 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 08:01:02.860427 21670 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:02.849392 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:35235)
0429 08:01:02.849789 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:02.849821 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:02.850583 (+ 762us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:02.851116 (+ 533us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:02.851124 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:02.851506 (+ 382us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:02.852349 (+ 843us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:02.852362 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.853166 (+ 804us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:02.853172 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:02.853554 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:02.853561 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:02.853781 (+ 220us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:02.854432 (+ 651us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:02.854456 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:02.855903 (+ 1447us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:02.858068 (+ 2165us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:02.858076 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:02.858087 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:02.858350 (+ 263us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:02.858658 (+ 308us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:02.858661 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:02.858663 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:02.858790 (+ 127us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:02.859152 (+ 362us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:02.859157 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:02.859363 (+ 206us) client_negotiation.cc:770] Sending connection context
0429 08:01:02.859553 (+ 190us) client_negotiation.cc:241] Negotiation successful
0429 08:01:02.859727 (+ 174us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260429 08:01:02.861395 21668 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34401
I20260429 08:01:02.861668 21668 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:02.862160 21668 heartbeater.cc:507] Master 127.12.12.190:34401 requested a full tablet report, sending...
I20260429 08:01:02.863123 21203 ts_manager.cc:194] Registered new tserver with Master: e86984f8e3394023a91521fa71743ebb (127.12.12.131:34669)
I20260429 08:01:02.863677 21203 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:35235
I20260429 08:01:02.870532 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.129.keytab.
Entry for principal oryx/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.129.keytab.
WARNING: no policy specified for oryx/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.130.keytab.
Entry for principal oryx/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.130.keytab.
WARNING: no policy specified for oryx/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.131.keytab.
Entry for principal oryx/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.131.keytab.
I20260429 08:01:02.941366 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21267
I20260429 08:01:02.947381 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21403
I20260429 08:01:02.953311 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21539
I20260429 08:01:02.959040 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21171
I20260429 08:01:02.966085 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34401
--webserver_interface=127.12.12.190
--webserver_port=33019
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:03.068167 21677 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:03.068419 21677 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:03.068502 21677 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:03.071717 21677 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:01:03.071810 21677 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:03.071853 21677 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:03.071913 21677 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:01:03.071957 21677 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:01:03.076128 21677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=33019
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:03.077286 21677 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:03.078171 21677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:03.083516 21683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:03.083621 21682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:03.083676 21677 server_base.cc:1061] running on GCE node
W20260429 08:01:03.083519 21685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:03.084225 21677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:03.085232 21677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:03.086396 21677 hybrid_clock.cc:648] HybridClock initialized: now 1777449663086365 us; error 46 us; skew 500 ppm
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449663, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:03.089123 21677 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:01:03.090050 21677 webserver.cc:492] Webserver started at http://127.12.12.190:33019/ using document root <none> and password file <none>
I20260429 08:01:03.090564 21677 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:03.090616 21677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:03.093863 21677 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:01:03.095621 21692 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.096578 21677 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:01:03.096668 21677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "52adb3700fb143edbd535e65baf28b2f"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "42932ca2f71a13b8db82eb5ece99f535"
server_key_iv: "4f1c1986738f1314b4195cf89feb3773"
server_key_version: "encryptionkey@0"
I20260429 08:01:03.097164 21677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:03.114987 21677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:03.117724 21677 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:03.117937 21677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:03.125379 21677 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:34401
I20260429 08:01:03.125413 21744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:34401 every 8 connection(s)
I20260429 08:01:03.126457 21677 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:01:03.130776 21745 sys_catalog.cc:263] Verifying existing consensus state
I20260429 08:01:03.132323 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Bootstrap starting.
I20260429 08:01:03.132344 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21677
I20260429 08:01:03.132897 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:36525
--local_ip_for_outbound_sockets=127.12.12.129
--tserver_master_addrs=127.12.12.190:34401
--webserver_port=45719
--webserver_interface=127.12.12.129
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.12.129
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 08:01:03.152627 21745 log.cc:826] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:03.160216 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 08:01:03.160674 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Bootstrap complete.
I20260429 08:01:03.165530 21745 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:03.165939 21745 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52adb3700fb143edbd535e65baf28b2f, State: Initialized, Role: FOLLOWER
I20260429 08:01:03.166478 21745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:03.166627 21745 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:01:03.166709 21745 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:01:03.166795 21745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 1 FOLLOWER]: Advancing to term 2
I20260429 08:01:03.168916 21745 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:03.169283 21745 leader_election.cc:304] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52adb3700fb143edbd535e65baf28b2f; no voters:
I20260429 08:01:03.169610 21745 leader_election.cc:290] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260429 08:01:03.169759 21749 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 2 FOLLOWER]: Leader election won for term 2
I20260429 08:01:03.170032 21749 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [term 2 LEADER]: Becoming Leader. State: Replica: 52adb3700fb143edbd535e65baf28b2f, State: Running, Role: LEADER
I20260429 08:01:03.170336 21749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } }
I20260429 08:01:03.170799 21745 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:01:03.171924 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52adb3700fb143edbd535e65baf28b2f. Latest consensus state: current_term: 2 leader_uuid: "52adb3700fb143edbd535e65baf28b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } } }
I20260429 08:01:03.171937 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52adb3700fb143edbd535e65baf28b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52adb3700fb143edbd535e65baf28b2f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34401 } } }
I20260429 08:01:03.172061 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:03.172061 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:03.173465 21758 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:01:03.176669 21758 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:01:03.177564 21758 catalog_manager.cc:1269] Loaded cluster ID: 8818857c55d84a6b837dca30f9f6c2ba
I20260429 08:01:03.177649 21758 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:01:03.179250 21758 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:01:03.179862 21758 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 52adb3700fb143edbd535e65baf28b2f: Loaded TSK: 0
I20260429 08:01:03.180617 21758 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 08:01:03.266273 21747 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:03.266538 21747 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:03.266631 21747 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:03.269928 21747 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:03.270035 21747 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:03.270148 21747 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:01:03.274602 21747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.129.keytab
--principal=oryx/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:36525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=45719
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:03.275701 21747 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:03.276557 21747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:03.283026 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:03.283016 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:03.283360 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:03.283852 21747 server_base.cc:1061] running on GCE node
I20260429 08:01:03.284217 21747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:03.284790 21747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:03.285985 21747 hybrid_clock.cc:648] HybridClock initialized: now 1777449663285965 us; error 36 us; skew 500 ppm
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449663, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:03.288546 21747 init.cc:377] Logged in from keytab as oryx/127.12.12.129@KRBTEST.COM (short username oryx)
I20260429 08:01:03.289702 21747 webserver.cc:492] Webserver started at http://127.12.12.129:45719/ using document root <none> and password file <none>
I20260429 08:01:03.290236 21747 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:03.290330 21747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:03.293705 21747 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:01:03.295585 21781 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.296582 21747 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:03.296718 21747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "c9bb06bb74f24505ad768c8eb15b2ad5"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "fd5a0ffe9e99f1db5db80287ccd858aa"
server_key_iv: "f3aecc8b4da731917b36eeb52749aaca"
server_key_version: "encryptionkey@0"
I20260429 08:01:03.297217 21747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:03.310029 21747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:03.312794 21747 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:03.313059 21747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:03.313654 21747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:03.314821 21747 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:03.314901 21747 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.314970 21747 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:03.315019 21747 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.326191 21747 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:36525
I20260429 08:01:03.326210 21894 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:36525 every 8 connection(s)
I20260429 08:01:03.327188 21747 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:01:03.329680 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21747
I20260429 08:01:03.330224 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:32965
--local_ip_for_outbound_sockets=127.12.12.130
--tserver_master_addrs=127.12.12.190:34401
--webserver_port=37363
--webserver_interface=127.12.12.130
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.12.130
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.344735 21898 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.329280 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:58079)
0429 08:01:03.333173 (+ 3893us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.333223 (+ 50us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.334306 (+ 1083us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.335693 (+ 1387us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.335705 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.336231 (+ 526us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.337498 (+ 1267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.337516 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.339301 (+ 1785us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.339305 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.339911 (+ 606us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.339922 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.340145 (+ 223us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.340894 (+ 749us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.340922 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.343876 (+ 2954us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3668,"thread_start_us":156,"threads_started":1}
W20260429 08:01:03.344740 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.329260 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:58079 (local address 127.12.12.190:34401)
0429 08:01:03.329914 (+ 654us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.329938 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.333363 (+ 3425us) server_negotiation.cc:408] Connection header received
0429 08:01:03.334834 (+ 1471us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.334851 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.335228 (+ 377us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.335591 (+ 363us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.337734 (+ 2143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.339139 (+ 1405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.340057 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.340361 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.344089 (+ 3728us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:58079: BlockingRecv error: recv got EOF from 127.12.12.129:58079 (error 108)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":128,"threads_started":1}
W20260429 08:01:03.345558 21895 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:34401 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.12.190:34401: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.351703 21898 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.346121 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:35669)
0429 08:01:03.346362 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.346376 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.346475 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.346756 (+ 281us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.346759 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.346925 (+ 166us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.347169 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.347176 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.348165 (+ 989us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.348169 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.348721 (+ 552us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.348730 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.348881 (+ 151us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.349500 (+ 619us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.349518 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.351565 (+ 2047us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260429 08:01:03.351859 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.346226 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:35669 (local address 127.12.12.190:34401)
0429 08:01:03.346379 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.346384 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.346425 (+ 41us) server_negotiation.cc:408] Connection header received
0429 08:01:03.346577 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.346580 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.346631 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.346713 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.347290 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.348040 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.348861 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.349044 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.351723 (+ 2679us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:35669: BlockingRecv error: recv got EOF from 127.12.12.129:35669 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.357784 21898 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.352431 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:47413)
0429 08:01:03.352595 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.352607 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.352697 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.352971 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.352975 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.353126 (+ 151us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.353324 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.353331 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.354341 (+ 1010us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.354344 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.354884 (+ 540us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.354895 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.355042 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.355549 (+ 507us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.355565 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.357635 (+ 2070us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260429 08:01:03.357893 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.352492 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:47413 (local address 127.12.12.190:34401)
0429 08:01:03.352595 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.352599 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.352667 (+ 68us) server_negotiation.cc:408] Connection header received
0429 08:01:03.352786 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.352789 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.352858 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.352938 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.353448 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.354221 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.355042 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.355554 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.357783 (+ 2229us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:47413: BlockingRecv error: recv got EOF from 127.12.12.129:47413 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260429 08:01:03.358263 21895 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 08:01:03.437134 21899 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:03.437386 21899 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:03.437482 21899 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:03.440619 21899 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:03.440730 21899 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:03.440855 21899 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:01:03.444974 21899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.130.keytab
--principal=oryx/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:32965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=37363
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:03.446046 21899 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:03.446890 21899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:03.453517 21904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:03.453836 21905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:03.454043 21907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:03.454087 21899 server_base.cc:1061] running on GCE node
I20260429 08:01:03.454408 21899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:03.454927 21899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:03.456073 21899 hybrid_clock.cc:648] HybridClock initialized: now 1777449663456061 us; error 25 us; skew 500 ppm
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449663, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:03.458662 21899 init.cc:377] Logged in from keytab as oryx/127.12.12.130@KRBTEST.COM (short username oryx)
I20260429 08:01:03.459674 21899 webserver.cc:492] Webserver started at http://127.12.12.130:37363/ using document root <none> and password file <none>
I20260429 08:01:03.460242 21899 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:03.460301 21899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:03.463797 21899 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:03.465685 21914 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.466648 21899 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260429 08:01:03.466799 21899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "e45743dae31d4d51a502e599ababcc30"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "893888853971d5775fba7b1ea61d55bc"
server_key_iv: "cd8dd648bc8507bdf3e12d71e30900e0"
server_key_version: "encryptionkey@0"
I20260429 08:01:03.467195 21899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:03.487578 21899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:03.490722 21899 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:03.490936 21899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:03.491531 21899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:03.492426 21899 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:03.492486 21899 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.492556 21899 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:03.492592 21899 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.502189 21899 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:32965
I20260429 08:01:03.502231 22027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:32965 every 8 connection(s)
I20260429 08:01:03.503232 21899 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:01:03.506269 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 21899
I20260429 08:01:03.506726 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:34669
--local_ip_for_outbound_sockets=127.12.12.131
--tserver_master_addrs=127.12.12.190:34401
--webserver_port=38161
--webserver_interface=127.12.12.131
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.12.131
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260429 08:01:03.513319 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.505035 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:60869 (local address 127.12.12.190:34401)
0429 08:01:03.505170 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.505174 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.505902 (+ 728us) server_negotiation.cc:408] Connection header received
0429 08:01:03.506846 (+ 944us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.506850 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.506916 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.507033 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.508641 (+ 1608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.509224 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.509947 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.510118 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.513197 (+ 3079us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:60869: BlockingRecv error: recv got EOF from 127.12.12.130:60869 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 08:01:03.513834 22030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.505279 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:60869)
0429 08:01:03.505740 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.505774 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.506634 (+ 860us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.507203 (+ 569us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.507211 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.507548 (+ 337us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.508479 (+ 931us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.508492 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.509348 (+ 856us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.509354 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.509836 (+ 482us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.509843 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.510046 (+ 203us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.510659 (+ 613us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.510677 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.513092 (+ 2415us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":108,"threads_started":1}
W20260429 08:01:03.514256 22028 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:34401 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.12.190:34401: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.520243 22030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.514747 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:44711)
0429 08:01:03.515149 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.515168 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.515271 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.515558 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.515562 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.515732 (+ 170us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.515997 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.516008 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.516691 (+ 683us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.516695 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.517359 (+ 664us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.517369 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.517511 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.518033 (+ 522us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.518048 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.520084 (+ 2036us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304}
W20260429 08:01:03.520524 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.514846 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:44711 (local address 127.12.12.190:34401)
0429 08:01:03.514968 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.514971 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.515233 (+ 262us) server_negotiation.cc:408] Connection header received
0429 08:01:03.515373 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.515377 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.515423 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.515489 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.516123 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.516571 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.517486 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.517702 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.520228 (+ 2526us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:44711: BlockingRecv error: recv got EOF from 127.12.12.130:44711 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.525607 22030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.521040 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:46943)
0429 08:01:03.521166 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.521180 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.521273 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.521606 (+ 333us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.521609 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.521731 (+ 122us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.521869 (+ 138us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.521873 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.522770 (+ 897us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.522773 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.523109 (+ 336us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.523115 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.523203 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.523754 (+ 551us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.523766 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.525480 (+ 1714us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260429 08:01:03.525799 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.521175 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:46943 (local address 127.12.12.190:34401)
0429 08:01:03.521330 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.521336 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.521349 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:01:03.521382 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.521385 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.521426 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.521510 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.521992 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.522656 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.523226 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.523407 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.525643 (+ 2236us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:46943: BlockingRecv error: recv got EOF from 127.12.12.130:46943 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260429 08:01:03.526012 22028 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 08:01:03.610117 22031 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:03.610358 22031 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:03.610440 22031 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:03.613646 22031 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:03.613759 22031 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:03.613864 22031 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:01:03.617839 22031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/krb5kdc/oryx_127.12.12.131.keytab
--principal=oryx/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:34669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=38161
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:03.618847 22031 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:03.619642 22031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:03.625767 22037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:03.625805 22036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:03.625964 22031 server_base.cc:1061] running on GCE node
W20260429 08:01:03.625788 22039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:03.626346 22031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:03.626843 22031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:03.627992 22031 hybrid_clock.cc:648] HybridClock initialized: now 1777449663627978 us; error 29 us; skew 500 ppm
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449663, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:03.630430 22031 init.cc:377] Logged in from keytab as oryx/127.12.12.131@KRBTEST.COM (short username oryx)
I20260429 08:01:03.631433 22031 webserver.cc:492] Webserver started at http://127.12.12.131:38161/ using document root <none> and password file <none>
I20260429 08:01:03.631932 22031 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:03.632014 22031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:03.635294 22031 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:03.637157 22046 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.638098 22031 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:03.638232 22031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "e86984f8e3394023a91521fa71743ebb"
format_stamp: "Formatted at 2026-04-29 08:01:02 on dist-test-slave-nhqz"
server_key: "dd785796d22a51c1f2a7987f2480913c"
server_key_iv: "ca0af4d3ddaaf82d94de6eaec9566430"
server_key_version: "encryptionkey@0"
I20260429 08:01:03.638648 22031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:03.653024 22031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:03.655452 22031 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:03.655670 22031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:03.656208 22031 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:03.657148 22031 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:03.657236 22031 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.657305 22031 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:03.657357 22031 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:03.666890 22031 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:34669
I20260429 08:01:03.666910 22159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:34669 every 8 connection(s)
I20260429 08:01:03.667910 22031 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:01:03.672588 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22031
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260429 08:01:03.678766 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.669806 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:39373 (local address 127.12.12.190:34401)
0429 08:01:03.669985 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.669990 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.670644 (+ 654us) server_negotiation.cc:408] Connection header received
0429 08:01:03.671511 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.671517 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.671615 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.671709 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.673701 (+ 1992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.674440 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.675290 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.675488 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.678631 (+ 3143us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:39373: BlockingRecv error: recv got EOF from 127.12.12.131:39373 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260429 08:01:03.679271 22162 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.670070 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:39373)
0429 08:01:03.670506 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.670538 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.671313 (+ 775us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.671865 (+ 552us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.671872 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.672289 (+ 417us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.673511 (+ 1222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.673528 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.674607 (+ 1079us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.674611 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.675188 (+ 577us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.675196 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.675344 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.675993 (+ 649us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.676010 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.678486 (+ 2476us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":92,"threads_started":1}
W20260429 08:01:03.679812 22160 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:34401 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.12.190:34401: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
I20260429 08:01:03.682468 22164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.673652 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.12.12.190:34401)
0429 08:01:03.673873 (+ 221us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.673876 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.673887 (+ 11us) server_negotiation.cc:408] Connection header received
0429 08:01:03.673916 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.673918 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.673955 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.674029 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.675037 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.675665 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.676394 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.676629 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.677796 (+ 1167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:03.677825 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:03.677838 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:03.677866 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:03.679946 (+ 2080us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:03.680575 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:03.680598 (+ 23us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:03.680603 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:03.680685 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:03.681004 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:03.681008 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:03.681010 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:03.681326 (+ 316us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:03.681534 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:03.681930 (+ 396us) server_negotiation.cc:300] Negotiation successful
0429 08:01:03.682288 (+ 358us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.685741 22162 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.680352 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:49559)
0429 08:01:03.680483 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.680497 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.680582 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.680861 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.680864 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.681018 (+ 154us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.681184 (+ 166us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.681188 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.681902 (+ 714us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.681907 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.682370 (+ 463us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.682378 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.682504 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.683677 (+ 1173us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.683692 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.685645 (+ 1953us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260429 08:01:03.685883 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.680408 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:49559 (local address 127.12.12.190:34401)
0429 08:01:03.680520 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.680526 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.680540 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:01:03.680693 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.680697 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.680742 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.680842 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.681313 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.681788 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.682501 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.682675 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.685770 (+ 3095us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:49559: BlockingRecv error: recv got EOF from 127.12.12.131:49559 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:03 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:03.690387 22162 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:03.686277 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:47521)
0429 08:01:03.686508 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:03.686521 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:03.686592 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:03.686819 (+ 227us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:03.686821 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:03.686936 (+ 115us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:03.687076 (+ 140us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.687080 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.687779 (+ 699us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.687782 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:03.688116 (+ 334us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:03.688121 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.688203 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.688659 (+ 456us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:03.688671 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:03.690298 (+ 1627us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167}
W20260429 08:01:03.690609 22160 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 08:01:03.690649 21897 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:03.686356 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:47521 (local address 127.12.12.190:34401)
0429 08:01:03.686508 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:03.686513 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:03.686559 (+ 46us) server_negotiation.cc:408] Connection header received
0429 08:01:03.686671 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:03.686673 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:03.686714 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:03.686795 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:03.687175 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.687684 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:03.688218 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:03.688367 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:03.690509 (+ 2142us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:47521: BlockingRecv error: recv got EOF from 127.12.12.131:47521 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Apr 29 08:01:04 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:04 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:04.364409 22165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:04.358975 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:58329)
0429 08:01:04.359287 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:04.359301 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:04.359436 (+ 135us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:04.359744 (+ 308us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:04.359747 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:04.360019 (+ 272us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:04.360246 (+ 227us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:04.360252 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.361051 (+ 799us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:04.361057 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:04.361495 (+ 438us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:04.361502 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.361627 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:04.362272 (+ 645us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:04.362285 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:04.364237 (+ 1952us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":129,"threads_started":1}
W20260429 08:01:04.364455 22166 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:04.359141 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:58329 (local address 127.12.12.190:34401)
0429 08:01:04.359372 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:04.359375 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:04.359390 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:01:04.359540 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:04.359543 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:04.359600 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:04.359726 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:04.360376 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.360947 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:04.361612 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.361867 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:04.364341 (+ 2474us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:58329: BlockingRecv error: recv got EOF from 127.12.12.129:58329 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":58,"threads_started":1}
Apr 29 08:01:04 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:04 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:04.531431 22167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:04.526653 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:33215)
0429 08:01:04.526956 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:04.526970 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:04.527075 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:04.527310 (+ 235us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:04.527312 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:04.527527 (+ 215us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:04.527769 (+ 242us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:04.527774 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.528473 (+ 699us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:04.528477 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:04.528932 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:04.528939 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.529035 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:04.529644 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:04.529657 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:04.531304 (+ 1647us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
W20260429 08:01:04.531538 22166 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:04.526789 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:33215 (local address 127.12.12.190:34401)
0429 08:01:04.526967 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:04.526970 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:04.527040 (+ 70us) server_negotiation.cc:408] Connection header received
0429 08:01:04.527155 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:04.527158 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:04.527206 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:04.527270 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:04.527869 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.528346 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:04.529024 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.529150 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:04.531431 (+ 2281us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:33215: BlockingRecv error: recv got EOF from 127.12.12.130:33215 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Apr 29 08:01:04 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:04 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:04.696323 22168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:04.691149 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:57873)
0429 08:01:04.691473 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:04.691486 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:04.691584 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:04.691822 (+ 238us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:04.691824 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:04.692046 (+ 222us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:04.692265 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:04.692271 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.693093 (+ 822us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:04.693097 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:04.693627 (+ 530us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:04.693639 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.693789 (+ 150us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:04.694446 (+ 657us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:04.694461 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:04.696200 (+ 1739us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
W20260429 08:01:04.696508 22166 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:04.691256 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:57873 (local address 127.12.12.190:34401)
0429 08:01:04.691417 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:04.691421 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:04.691555 (+ 134us) server_negotiation.cc:408] Connection header received
0429 08:01:04.691664 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:04.691667 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:04.691715 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:04.691786 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:04.692410 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.692996 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:04.693772 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:04.693992 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:04.696326 (+ 2334us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:57873: BlockingRecv error: recv got EOF from 127.12.12.131:57873 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Apr 29 08:01:05 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:05 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:05.370548 22169 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:05.365431 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:35199)
0429 08:01:05.365758 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:05.365772 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:05.365864 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:05.366141 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:05.366144 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:05.366315 (+ 171us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:05.366508 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:05.366513 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.367283 (+ 770us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:05.367286 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:05.367641 (+ 355us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:05.367645 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.367736 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:05.368333 (+ 597us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:05.368347 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:05.370385 (+ 2038us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
W20260429 08:01:05.370599 22170 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:05.365500 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:35199 (local address 127.12.12.190:34401)
0429 08:01:05.365818 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:05.365822 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:05.365837 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:01:05.365968 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:05.365971 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:05.366019 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:05.366110 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:05.366656 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.367171 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:05.367810 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.367971 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:05.370465 (+ 2494us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:35199: BlockingRecv error: recv got EOF from 127.12.12.129:35199 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
Apr 29 08:01:05 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:05 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:05.537657 22171 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:05.532372 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:48817)
0429 08:01:05.532654 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:05.532674 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:05.532785 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:05.533161 (+ 376us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:05.533164 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:05.533392 (+ 228us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:05.533599 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:05.533604 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.534406 (+ 802us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:05.534410 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:05.534852 (+ 442us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:05.534859 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.534979 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:05.535638 (+ 659us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:05.535655 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:05.537530 (+ 1875us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
W20260429 08:01:05.537787 22170 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:05.532609 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:48817 (local address 127.12.12.190:34401)
0429 08:01:05.532772 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:05.532775 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:05.532790 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:01:05.532959 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:05.532962 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:05.533009 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:05.533089 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:05.533723 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.534220 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:05.534958 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.535177 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:05.537668 (+ 2491us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:48817: BlockingRecv error: recv got EOF from 127.12.12.130:48817 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Apr 29 08:01:05 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:05 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:05.702752 22172 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:05.697233 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:50913)
0429 08:01:05.697552 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:05.697566 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:05.697675 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:05.697991 (+ 316us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:05.697994 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:05.698228 (+ 234us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:05.698514 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:05.698522 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.699565 (+ 1043us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:05.699568 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:05.699929 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:05.699935 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.700024 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:05.700588 (+ 564us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:05.700601 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:05.702624 (+ 2023us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
W20260429 08:01:05.702876 22170 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:05.697424 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:50913 (local address 127.12.12.190:34401)
0429 08:01:05.697645 (+ 221us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:05.697650 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:05.697670 (+ 20us) server_negotiation.cc:408] Connection header received
0429 08:01:05.697777 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:05.697780 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:05.697842 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:05.697940 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:05.698665 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.699458 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:05.700036 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:05.700172 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:05.702740 (+ 2568us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:50913: BlockingRecv error: recv got EOF from 127.12.12.131:50913 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Apr 29 08:01:06 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:06 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:06.376542 22174 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:06.371436 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:56453)
0429 08:01:06.371806 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:06.371819 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:06.371917 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:06.372163 (+ 246us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:06.372165 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:06.372376 (+ 211us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:06.372566 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:06.372571 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.373482 (+ 911us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:06.373487 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:06.373865 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:06.373871 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.373962 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:06.374520 (+ 558us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:06.374533 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:06.376424 (+ 1891us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
W20260429 08:01:06.376870 22173 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:06.371455 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:56453 (local address 127.12.12.190:34401)
0429 08:01:06.371762 (+ 307us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:06.371766 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:06.371896 (+ 130us) server_negotiation.cc:408] Connection header received
0429 08:01:06.371999 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:06.372002 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:06.372052 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:06.372130 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:06.372703 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.373318 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:06.373961 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.374100 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:06.376594 (+ 2494us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:56453: BlockingRecv error: recv got EOF from 127.12.12.129:56453 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
Apr 29 08:01:06 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:06 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:06.543727 22175 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:06.538507 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:50295)
0429 08:01:06.538779 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:06.538793 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:06.538897 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:06.539164 (+ 267us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:06.539166 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:06.539343 (+ 177us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:06.539541 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:06.539546 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.540354 (+ 808us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:06.540357 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:06.540727 (+ 370us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:06.540733 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.540858 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:06.541432 (+ 574us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:06.541445 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:06.543612 (+ 2167us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
W20260429 08:01:06.543841 22173 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:06.538597 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:50295 (local address 127.12.12.190:34401)
0429 08:01:06.538773 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:06.538776 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:06.538867 (+ 91us) server_negotiation.cc:408] Connection header received
0429 08:01:06.538980 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:06.538982 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:06.539047 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:06.539133 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:06.539679 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.540218 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:06.540912 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.541057 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:06.543724 (+ 2667us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:50295: BlockingRecv error: recv got EOF from 127.12.12.130:50295 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Apr 29 08:01:06 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:06 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:06.708881 22176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:06.703523 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:33009)
0429 08:01:06.703827 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:06.703841 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:06.703953 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:06.704224 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:06.704227 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:06.704466 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:06.704666 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:06.704672 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.705428 (+ 756us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:06.705432 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:06.705828 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:06.705834 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.705934 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:06.706547 (+ 613us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:06.706566 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:06.708699 (+ 2133us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
W20260429 08:01:06.708981 22173 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:06.703609 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:33009 (local address 127.12.12.190:34401)
0429 08:01:06.703754 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:06.703758 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:06.703914 (+ 156us) server_negotiation.cc:408] Connection header received
0429 08:01:06.704034 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:06.704037 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:06.704085 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:06.704160 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:06.704779 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.705306 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:06.705947 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:06.706125 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:06.708857 (+ 2732us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:33009: BlockingRecv error: recv got EOF from 127.12.12.131:33009 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 08:01:07 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:07 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:07.382699 22177 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:07.377441 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:60709)
0429 08:01:07.377774 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:07.377787 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:07.377892 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:07.378171 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:07.378174 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:07.378416 (+ 242us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:07.378621 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:07.378626 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.379362 (+ 736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:07.379364 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:07.379729 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:07.379735 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.379839 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:07.380484 (+ 645us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:07.380498 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:07.382518 (+ 2020us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":153,"threads_started":1}
W20260429 08:01:07.382752 22178 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:07.377538 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:60709 (local address 127.12.12.190:34401)
0429 08:01:07.377787 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:07.377790 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:07.377868 (+ 78us) server_negotiation.cc:408] Connection header received
0429 08:01:07.377970 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:07.377973 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:07.378020 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:07.378104 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:07.378730 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.379235 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:07.379848 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.380220 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:07.382636 (+ 2416us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:60709: BlockingRecv error: recv got EOF from 127.12.12.129:60709 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
Apr 29 08:01:07 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:07 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:07.550122 22179 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:07.544592 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:44917)
0429 08:01:07.544930 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:07.544944 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:07.545044 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:07.545361 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:07.545364 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:07.545632 (+ 268us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:07.545927 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:07.545936 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.547005 (+ 1069us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:07.547009 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:07.547456 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:07.547464 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.547572 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:07.548154 (+ 582us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:07.548168 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:07.550003 (+ 1835us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
W20260429 08:01:07.550240 22178 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:07.544661 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:44917 (local address 127.12.12.190:34401)
0429 08:01:07.544842 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:07.544846 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:07.545017 (+ 171us) server_negotiation.cc:408] Connection header received
0429 08:01:07.545126 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:07.545128 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:07.545178 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:07.545246 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:07.546083 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.546840 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:07.547564 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.547699 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:07.550125 (+ 2426us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:44917: BlockingRecv error: recv got EOF from 127.12.12.130:44917 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 08:01:07 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:07 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:07.714650 22180 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:07.709899 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:58845)
0429 08:01:07.710201 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:07.710213 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:07.710365 (+ 152us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:07.710709 (+ 344us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:07.710713 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:07.710920 (+ 207us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:07.711141 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:07.711148 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.711868 (+ 720us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:07.711870 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:07.712204 (+ 334us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:07.712208 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.712290 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:07.712859 (+ 569us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:07.712872 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:07.714524 (+ 1652us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
W20260429 08:01:07.714807 22178 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:07.709973 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:58845 (local address 127.12.12.190:34401)
0429 08:01:07.710117 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:07.710122 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:07.710290 (+ 168us) server_negotiation.cc:408] Connection header received
0429 08:01:07.710498 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:07.710501 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:07.710582 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:07.710668 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:07.711271 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.711767 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:07.712334 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:07.712529 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:07.714657 (+ 2128us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:58845: BlockingRecv error: recv got EOF from 127.12.12.131:58845 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Apr 29 08:01:08 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:08 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:08.392691 22181 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:08.383557 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:43169)
0429 08:01:08.383899 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:08.383914 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:08.384001 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:08.384251 (+ 250us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:08.384253 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:08.384496 (+ 243us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:08.384697 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:08.384702 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.385415 (+ 713us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:08.385418 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:08.385794 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:08.385800 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.385890 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:08.390386 (+ 4496us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:08.390400 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:08.392517 (+ 2117us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":132,"threads_started":1}
W20260429 08:01:08.392755 22182 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:08.383710 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:43169 (local address 127.12.12.190:34401)
0429 08:01:08.383943 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:08.383946 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:08.383962 (+ 16us) server_negotiation.cc:408] Connection header received
0429 08:01:08.384093 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:08.384096 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:08.384139 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:08.384215 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:08.384821 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.385320 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:08.385891 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.386102 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:08.392642 (+ 6540us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:43169: BlockingRecv error: recv got EOF from 127.12.12.129:43169 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":58,"threads_started":1}
Apr 29 08:01:08 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:08 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:08.555546 22183 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:08.550917 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:33399)
0429 08:01:08.551200 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:08.551213 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:08.551303 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:08.551520 (+ 217us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:08.551523 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:08.551730 (+ 207us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:08.551919 (+ 189us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:08.551924 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.552631 (+ 707us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:08.552633 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:08.553041 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:08.553047 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.553150 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:08.553739 (+ 589us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:08.553752 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:08.555437 (+ 1685us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
W20260429 08:01:08.555714 22182 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:08.551061 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:33399 (local address 127.12.12.190:34401)
0429 08:01:08.551194 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:08.551198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:08.551280 (+ 82us) server_negotiation.cc:408] Connection header received
0429 08:01:08.551369 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:08.551371 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:08.551415 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:08.551481 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:08.552019 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.552522 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:08.553150 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.553316 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:08.555564 (+ 2248us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:33399: BlockingRecv error: recv got EOF from 127.12.12.130:33399 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 29 08:01:08 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:08 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:08.720690 22184 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:08.715386 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:40953)
0429 08:01:08.715757 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:08.715771 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:08.715908 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:08.716206 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:08.716209 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:08.716451 (+ 242us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:08.716642 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:08.716647 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.717443 (+ 796us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:08.717449 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:08.717843 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:08.717849 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.717977 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:08.718645 (+ 668us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:08.718659 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:08.720519 (+ 1860us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":91,"threads_started":1}
W20260429 08:01:08.720757 22182 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:08.715521 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:40953 (local address 127.12.12.190:34401)
0429 08:01:08.715704 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:08.715709 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:08.715854 (+ 145us) server_negotiation.cc:408] Connection header received
0429 08:01:08.716010 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:08.716014 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:08.716076 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:08.716170 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:08.716769 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.717329 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:08.717959 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:08.718159 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:08.720649 (+ 2490us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:40953: BlockingRecv error: recv got EOF from 127.12.12.131:40953 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Apr 29 08:01:09 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:09 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:09.399034 22185 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:09.393577 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:47459)
0429 08:01:09.393873 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:09.393889 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:09.393989 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:09.394286 (+ 297us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:09.394290 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:09.394475 (+ 185us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:09.394679 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:09.394685 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.395609 (+ 924us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:09.395614 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:09.396151 (+ 537us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:09.396160 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.396281 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:09.396966 (+ 685us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:09.396988 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:09.398868 (+ 1880us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
W20260429 08:01:09.399102 22186 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:09.393671 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:47459 (local address 127.12.12.190:34401)
0429 08:01:09.393958 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:09.393962 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:09.393978 (+ 16us) server_negotiation.cc:408] Connection header received
0429 08:01:09.394102 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:09.394105 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:09.394153 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:09.394232 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:09.394793 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.395453 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:09.396316 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.396597 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:09.398986 (+ 2389us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:47459: BlockingRecv error: recv got EOF from 127.12.12.129:47459 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
Apr 29 08:01:09 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:09 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:09.561401 22187 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:09.556305 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:43891)
0429 08:01:09.556617 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:09.556631 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:09.556729 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:09.557063 (+ 334us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:09.557066 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:09.557242 (+ 176us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:09.557432 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:09.557436 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.558209 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:09.558212 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:09.558583 (+ 371us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:09.558588 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.558688 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:09.559215 (+ 527us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:09.559227 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:09.561196 (+ 1969us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":133,"threads_started":1}
W20260429 08:01:09.561400 22186 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:09.556464 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:43891 (local address 127.12.12.190:34401)
0429 08:01:09.556614 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:09.556618 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:09.556734 (+ 116us) server_negotiation.cc:408] Connection header received
0429 08:01:09.556829 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:09.556832 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:09.556933 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:09.557028 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:09.557558 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.558073 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:09.558711 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.558854 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:09.561307 (+ 2453us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:43891: BlockingRecv error: recv got EOF from 127.12.12.130:43891 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Apr 29 08:01:09 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:09 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:09.726708 22188 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:09.721674 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:57083)
0429 08:01:09.721997 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:09.722011 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:09.722120 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:09.722367 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:09.722370 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:09.722675 (+ 305us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:09.722888 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:09.722895 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.723640 (+ 745us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:09.723643 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:09.724021 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:09.724027 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.724128 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:09.724786 (+ 658us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:09.724799 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:09.726594 (+ 1795us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":142,"threads_started":1}
W20260429 08:01:09.726836 22186 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:09.721787 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:57083 (local address 127.12.12.190:34401)
0429 08:01:09.721922 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:09.721925 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:09.722084 (+ 159us) server_negotiation.cc:408] Connection header received
0429 08:01:09.722202 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:09.722205 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:09.722253 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:09.722327 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:09.723011 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.723511 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:09.724140 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:09.724291 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:09.726715 (+ 2424us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:57083: BlockingRecv error: recv got EOF from 127.12.12.131:57083 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Apr 29 08:01:10 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:10 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:10.405311 22189 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:10.399902 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:55477)
0429 08:01:10.400228 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:10.400242 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:10.400356 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:10.400703 (+ 347us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:10.400705 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:10.400923 (+ 218us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:10.401234 (+ 311us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:10.401242 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.402011 (+ 769us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:10.402015 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:10.402495 (+ 480us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:10.402502 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.402612 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:10.403134 (+ 522us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:10.403147 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:10.405131 (+ 1984us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":89,"threads_started":1}
W20260429 08:01:10.405421 22190 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:10.399987 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:55477 (local address 127.12.12.190:34401)
0429 08:01:10.400228 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:10.400231 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:10.400315 (+ 84us) server_negotiation.cc:408] Connection header received
0429 08:01:10.400487 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:10.400490 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:10.400542 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:10.400684 (+ 142us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:10.401374 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.401895 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:10.402625 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.402798 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:10.405295 (+ 2497us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:55477: BlockingRecv error: recv got EOF from 127.12.12.129:55477 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
Apr 29 08:01:10 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:10 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:10.566972 22191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:10.562148 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:51555)
0429 08:01:10.562437 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:10.562451 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:10.562551 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:10.562852 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:10.562854 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:10.563028 (+ 174us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:10.563231 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:10.563236 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.563936 (+ 700us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:10.563938 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:10.564310 (+ 372us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:10.564315 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.564406 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:10.565003 (+ 597us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:10.565015 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:10.566857 (+ 1842us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
W20260429 08:01:10.567113 22190 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:10.562277 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:51555 (local address 127.12.12.190:34401)
0429 08:01:10.562453 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:10.562456 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:10.562522 (+ 66us) server_negotiation.cc:408] Connection header received
0429 08:01:10.562687 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:10.562689 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:10.562741 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:10.562815 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:10.563340 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.563842 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:10.564406 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.564558 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:10.566991 (+ 2433us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:51555: BlockingRecv error: recv got EOF from 127.12.12.130:51555 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Apr 29 08:01:10 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:10 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:10.732940 22192 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:10.727580 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:55539)
0429 08:01:10.727880 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:10.727893 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:10.727993 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:10.728302 (+ 309us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:10.728304 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:10.728481 (+ 177us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:10.728680 (+ 199us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:10.728685 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.729472 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:10.729475 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:10.729855 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:10.729860 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.729949 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:10.730484 (+ 535us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:10.730513 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:10.732738 (+ 2225us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
W20260429 08:01:10.732977 22190 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:10.727663 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:55539 (local address 127.12.12.190:34401)
0429 08:01:10.727816 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:10.727819 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:10.727966 (+ 147us) server_negotiation.cc:408] Connection header received
0429 08:01:10.728114 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:10.728118 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:10.728169 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:10.728268 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:10.728791 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.729376 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:10.729949 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:10.730091 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:10.732881 (+ 2790us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:55539: BlockingRecv error: recv got EOF from 127.12.12.131:55539 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 08:01:11 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:11 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:11.411355 22193 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:11.406462 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:34971)
0429 08:01:11.406751 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:11.406764 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:11.406850 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:11.407088 (+ 238us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:11.407091 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:11.407266 (+ 175us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:11.407458 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:11.407462 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.408216 (+ 754us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:11.408219 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:11.408600 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:11.408605 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.408691 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:11.409323 (+ 632us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:11.409337 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:11.411182 (+ 1845us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":109,"threads_started":1}
W20260429 08:01:11.411437 22194 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:11.406593 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:34971 (local address 127.12.12.190:34401)
0429 08:01:11.406869 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:11.406873 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:11.406886 (+ 13us) server_negotiation.cc:408] Connection header received
0429 08:01:11.406920 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:11.406922 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:11.406967 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:11.407053 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:11.407567 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.408119 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:11.408703 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.409023 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:11.411309 (+ 2286us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:34971: BlockingRecv error: recv got EOF from 127.12.12.129:34971 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":129,"threads_started":1}
Apr 29 08:01:11 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:11 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:11.573133 22195 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:11.567732 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:46779)
0429 08:01:11.568014 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:11.568026 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:11.568124 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:11.568510 (+ 386us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:11.568513 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:11.568699 (+ 186us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:11.568965 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:11.568972 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.569846 (+ 874us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:11.569849 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:11.570211 (+ 362us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:11.570216 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.570305 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:11.570964 (+ 659us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:11.570976 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:11.573016 (+ 2040us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
W20260429 08:01:11.573314 22194 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:11.567802 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:46779 (local address 127.12.12.190:34401)
0429 08:01:11.567956 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:11.567960 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:11.568121 (+ 161us) server_negotiation.cc:408] Connection header received
0429 08:01:11.568332 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:11.568335 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:11.568390 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:11.568470 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:11.569072 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.569755 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:11.570313 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.570551 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:11.573156 (+ 2605us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:46779: BlockingRecv error: recv got EOF from 127.12.12.130:46779 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Apr 29 08:01:11 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:11 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:11.738767 22196 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:11.733766 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:34281)
0429 08:01:11.734078 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:11.734091 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:11.734228 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:11.734507 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:11.734510 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:11.734723 (+ 213us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:11.734923 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:11.734929 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.735685 (+ 756us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:11.735691 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:11.736085 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:11.736091 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.736232 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:11.736835 (+ 603us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:11.736849 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:11.738643 (+ 1794us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
W20260429 08:01:11.738945 22194 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:11.733833 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:34281 (local address 127.12.12.190:34401)
0429 08:01:11.734019 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:11.734023 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:11.734168 (+ 145us) server_negotiation.cc:408] Connection header received
0429 08:01:11.734340 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:11.734344 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:11.734397 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:11.734484 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:11.735074 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.735574 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:11.736216 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:11.736425 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:11.738807 (+ 2382us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:34281: BlockingRecv error: recv got EOF from 127.12.12.131:34281 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Apr 29 08:01:12 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:12 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:12.417824 22197 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:12.412504 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:46735)
0429 08:01:12.412801 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:12.412851 (+ 50us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:12.412979 (+ 128us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:12.413289 (+ 310us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:12.413293 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:12.413551 (+ 258us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:12.413820 (+ 269us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:12.413829 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.414553 (+ 724us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:12.414556 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:12.414940 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:12.414945 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.415035 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:12.415584 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:12.415596 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:12.417708 (+ 2112us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
W20260429 08:01:12.417987 22198 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:12.412596 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:46735 (local address 127.12.12.190:34401)
0429 08:01:12.412893 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:12.412897 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:12.412913 (+ 16us) server_negotiation.cc:408] Connection header received
0429 08:01:12.413086 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:12.413092 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:12.413150 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:12.413238 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:12.413935 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.414448 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:12.415038 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.415180 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:12.417846 (+ 2666us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:46735: BlockingRecv error: recv got EOF from 127.12.12.129:46735 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
Apr 29 08:01:12 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:12 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:12.579250 22199 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:12.574142 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:41003)
0429 08:01:12.574467 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:12.574495 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:12.574596 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:12.574830 (+ 234us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:12.574832 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:12.575004 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:12.575192 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:12.575197 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.575888 (+ 691us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:12.575891 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:12.576299 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:12.576306 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.576441 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:12.577109 (+ 668us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:12.577126 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:12.579122 (+ 1996us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":147,"threads_started":1}
W20260429 08:01:12.579505 22198 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:12.574187 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41003 (local address 127.12.12.190:34401)
0429 08:01:12.574329 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:12.574332 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:12.574573 (+ 241us) server_negotiation.cc:408] Connection header received
0429 08:01:12.574664 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:12.574666 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:12.574726 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:12.574796 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:12.575299 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.575791 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:12.576427 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.576643 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:12.579404 (+ 2761us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:41003: BlockingRecv error: recv got EOF from 127.12.12.130:41003 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 29 08:01:12 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:12 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:12.744992 22200 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:12.739696 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:60831)
0429 08:01:12.740043 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:12.740064 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:12.740206 (+ 142us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:12.740504 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:12.740507 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:12.740757 (+ 250us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:12.741055 (+ 298us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:12.741063 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.741884 (+ 821us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:12.741889 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:12.742277 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:12.742283 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.742415 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:12.743021 (+ 606us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:12.743043 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:12.744803 (+ 1760us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":93,"threads_started":1}
W20260429 08:01:12.745064 22198 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:12.739849 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:60831 (local address 127.12.12.190:34401)
0429 08:01:12.740006 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:12.740011 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:12.740140 (+ 129us) server_negotiation.cc:408] Connection header received
0429 08:01:12.740327 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:12.740330 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:12.740389 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:12.740482 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:12.741183 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.741762 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:12.742396 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:12.742593 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:12.744953 (+ 2360us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:60831: BlockingRecv error: recv got EOF from 127.12.12.131:60831 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 08:01:13 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:13 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:13.428016 22201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:13.418945 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:53983)
0429 08:01:13.419263 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:13.419282 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:13.419399 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:13.419714 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:13.419716 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:13.419949 (+ 233us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:13.420219 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:13.420230 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.421100 (+ 870us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:13.421103 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:13.421524 (+ 421us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:13.421530 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.421626 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:13.425877 (+ 4251us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:13.425897 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:13.427893 (+ 1996us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
W20260429 08:01:13.428274 22202 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:13.418981 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:53983 (local address 127.12.12.190:34401)
0429 08:01:13.419389 (+ 408us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:13.419417 (+ 28us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:13.419435 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:01:13.419482 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:13.419486 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:13.419545 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:13.419675 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:13.420367 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.421001 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:13.421634 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.421778 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:13.428073 (+ 6295us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:53983: BlockingRecv error: recv got EOF from 127.12.12.129:53983 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":79,"threads_started":1}
Apr 29 08:01:13 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:13 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:13.585350 22203 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:13.580270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:36071)
0429 08:01:13.580585 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:13.580599 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:13.580697 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:13.581008 (+ 311us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:13.581011 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:13.581194 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:13.581403 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:13.581408 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.582126 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:13.582129 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:13.582555 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:13.582561 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.582655 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:13.583253 (+ 598us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:13.583267 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:13.585206 (+ 1939us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
W20260429 08:01:13.585577 22202 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:13.580392 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:36071 (local address 127.12.12.190:34401)
0429 08:01:13.580560 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:13.580565 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:13.580674 (+ 109us) server_negotiation.cc:408] Connection header received
0429 08:01:13.580782 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:13.580785 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:13.580859 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:13.580939 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:13.581521 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.582027 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:13.582655 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.582785 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:13.585390 (+ 2605us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:36071: BlockingRecv error: recv got EOF from 127.12.12.130:36071 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 08:01:13 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:13 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:13.751245 22204 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:13.745836 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:49259)
0429 08:01:13.746185 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:13.746202 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:13.746393 (+ 191us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:13.746671 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:13.746674 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:13.746887 (+ 213us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:13.747075 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:13.747080 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.747824 (+ 744us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:13.747831 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:13.748322 (+ 491us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:13.748329 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.748562 (+ 233us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:13.749154 (+ 592us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:13.749167 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:13.751074 (+ 1907us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
W20260429 08:01:13.751308 22202 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:13.746023 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:49259 (local address 127.12.12.190:34401)
0429 08:01:13.746186 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:13.746192 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:13.746329 (+ 137us) server_negotiation.cc:408] Connection header received
0429 08:01:13.746500 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:13.746503 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:13.746560 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:13.746650 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:13.747201 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.747713 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:13.748539 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:13.748765 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:13.751192 (+ 2427us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:49259: BlockingRecv error: recv got EOF from 127.12.12.131:49259 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Apr 29 08:01:14 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:14 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:14.433864 22206 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:14.428911 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:46351)
0429 08:01:14.429239 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:14.429252 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:14.429366 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:14.429643 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:14.429645 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:14.429817 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:14.430007 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:14.430011 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.430814 (+ 803us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:14.430816 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:14.431183 (+ 367us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:14.431188 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.431273 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:14.431882 (+ 609us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:14.431896 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:14.433739 (+ 1843us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
W20260429 08:01:14.434031 22205 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:14.428956 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:46351 (local address 127.12.12.190:34401)
0429 08:01:14.429257 (+ 301us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:14.429263 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:14.429315 (+ 52us) server_negotiation.cc:408] Connection header received
0429 08:01:14.429473 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:14.429476 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:14.429530 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:14.429611 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:14.430125 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.430716 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:14.431286 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.431443 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:14.433880 (+ 2437us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:46351: BlockingRecv error: recv got EOF from 127.12.12.129:46351 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
Apr 29 08:01:14 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:14 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:14.591404 22207 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:14.586411 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:59409)
0429 08:01:14.586757 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:14.586771 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:14.586872 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:14.587169 (+ 297us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:14.587171 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:14.587350 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:14.587592 (+ 242us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:14.587599 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.588616 (+ 1017us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:14.588619 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:14.589014 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:14.589020 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.589108 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:14.589638 (+ 530us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:14.589651 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:14.591296 (+ 1645us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":84,"threads_started":1}
W20260429 08:01:14.591536 22205 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:14.586535 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:59409 (local address 127.12.12.190:34401)
0429 08:01:14.586698 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:14.586701 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:14.586842 (+ 141us) server_negotiation.cc:408] Connection header received
0429 08:01:14.586965 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:14.586968 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:14.587039 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:14.587139 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:14.587732 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.588498 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:14.589106 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.589242 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:14.591422 (+ 2180us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:59409: BlockingRecv error: recv got EOF from 127.12.12.130:59409 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Apr 29 08:01:14 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:14 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:14.757280 22208 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:14.752163 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:35193)
0429 08:01:14.752543 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:14.752558 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:14.752695 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:14.753000 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:14.753004 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:14.753237 (+ 233us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:14.753497 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:14.753504 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.754212 (+ 708us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:14.754218 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:14.754615 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:14.754620 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.754744 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:14.755267 (+ 523us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:14.755279 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:14.757115 (+ 1836us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":157,"threads_started":1}
W20260429 08:01:14.757331 22205 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:14.752357 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:35193 (local address 127.12.12.190:34401)
0429 08:01:14.752498 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:14.752502 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:14.752666 (+ 164us) server_negotiation.cc:408] Connection header received
0429 08:01:14.752838 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:14.752842 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:14.752889 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:14.752979 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:14.753619 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.754101 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:14.754729 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:14.754928 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:14.757231 (+ 2303us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:35193: BlockingRecv error: recv got EOF from 127.12.12.131:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Apr 29 08:01:15 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:15 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:15.440099 22209 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:15.434730 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:44069)
0429 08:01:15.435075 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:15.435090 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:15.435235 (+ 145us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:15.435536 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:15.435538 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:15.435725 (+ 187us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:15.435925 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:15.435930 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.436717 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:15.436722 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:15.437247 (+ 525us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:15.437257 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.437370 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:15.437977 (+ 607us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:15.437992 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:15.439931 (+ 1939us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
W20260429 08:01:15.440155 22210 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:15.434855 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:44069 (local address 127.12.12.190:34401)
0429 08:01:15.435101 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:15.435105 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:15.435168 (+ 63us) server_negotiation.cc:408] Connection header received
0429 08:01:15.435334 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:15.435336 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:15.435382 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:15.435460 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:15.436031 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.436549 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:15.437366 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.437505 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:15.440046 (+ 2541us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:44069: BlockingRecv error: recv got EOF from 127.12.12.129:44069 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":59,"threads_started":1}
Apr 29 08:01:15 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:15 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:15.597720 22211 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:15.592258 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:33687)
0429 08:01:15.592530 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:15.592543 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:15.592644 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:15.592904 (+ 260us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:15.592906 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:15.593084 (+ 178us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:15.593290 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:15.593296 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.594200 (+ 904us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:15.594204 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:15.594593 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:15.594600 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.594699 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:15.595228 (+ 529us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:15.595241 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:15.597593 (+ 2352us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
W20260429 08:01:15.597790 22210 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:15.592312 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:33687 (local address 127.12.12.190:34401)
0429 08:01:15.592477 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:15.592480 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:15.592614 (+ 134us) server_negotiation.cc:408] Connection header received
0429 08:01:15.592725 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:15.592728 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:15.592776 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:15.592867 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:15.593428 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.593940 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:15.594716 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.594866 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:15.597697 (+ 2831us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:33687: BlockingRecv error: recv got EOF from 127.12.12.130:33687 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Apr 29 08:01:15 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:15 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:15.763571 22212 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:15.758264 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:45511)
0429 08:01:15.758636 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:15.758651 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:15.758796 (+ 145us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:15.759062 (+ 266us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:15.759065 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:15.759288 (+ 223us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:15.759499 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:15.759505 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.760392 (+ 887us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:15.760396 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:15.760902 (+ 506us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:15.760911 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.761034 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:15.761622 (+ 588us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:15.761636 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:15.763442 (+ 1806us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":96,"threads_started":1}
W20260429 08:01:15.763777 22210 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:15.758419 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:45511 (local address 127.12.12.190:34401)
0429 08:01:15.758609 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:15.758614 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:15.758732 (+ 118us) server_negotiation.cc:408] Connection header received
0429 08:01:15.758897 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:15.758900 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:15.758953 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:15.759041 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:15.759642 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.760262 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:15.761033 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:15.761291 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:15.763621 (+ 2330us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:45511: BlockingRecv error: recv got EOF from 127.12.12.131:45511 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Apr 29 08:01:16 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:16 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:16.446118 22213 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:16.441011 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:51761)
0429 08:01:16.441286 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:16.441301 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:16.441399 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:16.441655 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:16.441657 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:16.441833 (+ 176us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:16.442026 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:16.442031 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.442763 (+ 732us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:16.442765 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:16.443216 (+ 451us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:16.443224 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.443314 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:16.443870 (+ 556us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:16.443883 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:16.445946 (+ 2063us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20260429 08:01:16.446175 22214 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:16.441068 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:51761 (local address 127.12.12.190:34401)
0429 08:01:16.441344 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:16.441348 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:16.441362 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:01:16.441493 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:16.441495 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:16.441541 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:16.441621 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:16.442129 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.442667 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:16.443340 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.443491 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:16.446056 (+ 2565us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:51761: BlockingRecv error: recv got EOF from 127.12.12.129:51761 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":63,"threads_started":1}
Apr 29 08:01:16 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:16 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:16.603770 22215 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:16.598711 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:58229)
0429 08:01:16.599029 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:16.599042 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:16.599132 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:16.599354 (+ 222us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:16.599357 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:16.599543 (+ 186us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:16.599731 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:16.599736 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.600483 (+ 747us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:16.600486 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:16.600989 (+ 503us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:16.600996 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.601084 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:16.601737 (+ 653us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:16.601750 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:16.603631 (+ 1881us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260429 08:01:16.603870 22214 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:16.598997 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:58229 (local address 127.12.12.190:34401)
0429 08:01:16.599125 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:16.599129 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:16.599141 (+ 12us) server_negotiation.cc:408] Connection header received
0429 08:01:16.599202 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:16.599204 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:16.599249 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:16.599316 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:16.599850 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.600373 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:16.601092 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.601315 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:16.603751 (+ 2436us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:58229: BlockingRecv error: recv got EOF from 127.12.12.130:58229 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Apr 29 08:01:16 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:16 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:16.769873 22216 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:16.764431 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:33637)
0429 08:01:16.764736 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:16.764748 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:16.764951 (+ 203us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:16.765228 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:16.765231 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:16.765452 (+ 221us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:16.765665 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:16.765671 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.766401 (+ 730us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:16.766407 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:16.766792 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:16.766797 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.766926 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:16.767644 (+ 718us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:16.767657 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:16.769758 (+ 2101us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
W20260429 08:01:16.769999 22214 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:16.764505 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:33637 (local address 127.12.12.190:34401)
0429 08:01:16.764672 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:16.764677 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:16.764851 (+ 174us) server_negotiation.cc:408] Connection header received
0429 08:01:16.765055 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:16.765061 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:16.765115 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:16.765205 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:16.765792 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.766289 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:16.766909 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:16.767117 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:16.769881 (+ 2764us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:33637: BlockingRecv error: recv got EOF from 127.12.12.131:33637 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Apr 29 08:01:17 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:17 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:17.452018 22217 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:17.446919 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:32925)
0429 08:01:17.447225 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:17.447239 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:17.447331 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:17.447561 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:17.447563 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:17.447735 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:17.447958 (+ 223us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:17.447966 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.449032 (+ 1066us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:17.449036 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:17.449543 (+ 507us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:17.449551 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.449652 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:17.450151 (+ 499us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:17.450164 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:17.451908 (+ 1744us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":126,"threads_started":1}
W20260429 08:01:17.452155 22218 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:17.447056 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:32925 (local address 127.12.12.190:34401)
0429 08:01:17.447338 (+ 282us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:17.447342 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:17.447356 (+ 14us) server_negotiation.cc:408] Connection header received
0429 08:01:17.447394 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:17.447396 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:17.447441 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:17.447527 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:17.448123 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.448898 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:17.449651 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.449792 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:17.452030 (+ 2238us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:32925: BlockingRecv error: recv got EOF from 127.12.12.129:32925 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
Apr 29 08:01:17 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:17 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:17.609367 22219 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:17.604520 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:50289)
0429 08:01:17.604784 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:17.604798 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:17.604915 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:17.605182 (+ 267us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:17.605184 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:17.605377 (+ 193us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:17.605567 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:17.605572 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.606350 (+ 778us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:17.606355 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:17.606757 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:17.606763 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.606894 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:17.607460 (+ 566us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:17.607473 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:17.609215 (+ 1742us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260429 08:01:17.609480 22218 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:17.604617 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:50289 (local address 127.12.12.190:34401)
0429 08:01:17.604847 (+ 230us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:17.604854 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:17.604871 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:01:17.605022 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:17.605027 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:17.605080 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:17.605150 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:17.605681 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.606175 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:17.606880 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.607090 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:17.609371 (+ 2281us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:50289: BlockingRecv error: recv got EOF from 127.12.12.130:50289 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Apr 29 08:01:17 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:17 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.131@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:17.776095 22220 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:17.770725 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.131:60441)
0429 08:01:17.771011 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:17.771046 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:17.771158 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:17.771489 (+ 331us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:17.771492 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:17.771697 (+ 205us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:17.771907 (+ 210us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:17.771913 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.772848 (+ 935us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:17.772854 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:17.773315 (+ 461us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:17.773323 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.773452 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:17.774062 (+ 610us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:17.774076 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:17.775976 (+ 1900us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
W20260429 08:01:17.776288 22218 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:17.770954 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:60441 (local address 127.12.12.190:34401)
0429 08:01:17.771122 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:17.771127 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:17.771142 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:01:17.771324 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:17.771327 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:17.771375 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:17.771468 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:17.772033 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.772689 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:17.773432 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:17.773653 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:17.776117 (+ 2464us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.131:60441: BlockingRecv error: recv got EOF from 127.12.12.131:60441 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Apr 29 08:01:18 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:18 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.129@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260429 08:01:18.460619 22221 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:18.452902 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.129:43329)
0429 08:01:18.453377 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:18.453392 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:18.453493 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:18.453713 (+ 220us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:18.453715 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:18.453877 (+ 162us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:18.454061 (+ 184us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:18.454066 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.454768 (+ 702us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:18.454770 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:18.455125 (+ 355us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:18.455130 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.455212 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:18.458595 (+ 3383us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:18.458611 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:18.460441 (+ 1830us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":95,"threads_started":1}
W20260429 08:01:18.460681 22222 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:18.453187 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:43329 (local address 127.12.12.190:34401)
0429 08:01:18.453410 (+ 223us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:18.453414 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:18.453446 (+ 32us) server_negotiation.cc:408] Connection header received
0429 08:01:18.453568 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:18.453570 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:18.453610 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:18.453684 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:18.454159 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.454675 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:18.455213 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.455345 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:18.460570 (+ 5225us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.129:43329: BlockingRecv error: recv got EOF from 127.12.12.129:43329 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
Apr 29 08:01:18 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
Apr 29 08:01:18 dist-test-slave-nhqz krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.12.130@KRBTEST.COM for oryx/127.12.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260429 08:01:18.615756 22222 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:18.610529 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:32771 (local address 127.12.12.190:34401)
0429 08:01:18.610690 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:18.610695 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:18.610810 (+ 115us) server_negotiation.cc:408] Connection header received
0429 08:01:18.610995 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:18.610998 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:18.611052 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:18.611140 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:18.611699 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.612198 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:18.612786 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.613053 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:18.615635 (+ 2582us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.12.130:32771: BlockingRecv error: recv got EOF from 127.12.12.130:32771 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 08:01:18.615864 22223 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:18.610448 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34401 (local address 127.12.12.130:32771)
0429 08:01:18.610741 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:18.610754 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:18.610853 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:18.611165 (+ 312us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:18.611167 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:18.611342 (+ 175us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:18.611565 (+ 223us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:18.611571 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.612305 (+ 734us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:18.612307 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:18.612690 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:18.612695 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.612784 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:18.613388 (+ 604us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:18.613401 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:18.615501 (+ 2100us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.12.190:34401: Server oryx/127.12.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260429 08:01:18.673918 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21747
I20260429 08:01:18.679781 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21899
I20260429 08:01:18.685400 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22031
I20260429 08:01:18.690937 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 21677
2026-04-29T08:01:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18853 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-04-29T08:01:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:01:18Z Disabled control of system clock
I20260429 08:01:18.720690 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:45901
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:45901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260429 08:01:18.821730 22227 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:18.821954 22227 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:18.821996 22227 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:18.825354 22227 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:01:18.825413 22227 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:18.825429 22227 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:18.825448 22227 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:01:18.825464 22227 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:01:18.829223 22227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:45901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:45901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:18.830188 22227 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:18.831244 22227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 08:01:18.837002 22227 server_base.cc:1061] running on GCE node
W20260429 08:01:18.836916 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:18.836920 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:18.836935 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:18.837710 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:18.838562 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:18.839716 22227 hybrid_clock.cc:648] HybridClock initialized: now 1777449678839699 us; error 30 us; skew 500 ppm
I20260429 08:01:18.841562 22227 webserver.cc:492] Webserver started at http://127.12.12.190:35115/ using document root <none> and password file <none>
I20260429 08:01:18.842108 22227 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:18.842164 22227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:18.842370 22227 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:18.843997 22227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "124e707237ff49239814996cef82e464"
format_stamp: "Formatted at 2026-04-29 08:01:18 on dist-test-slave-nhqz"
server_key: "396a60e02b8c20609b32274894953780"
server_key_iv: "a8952d18209a32d4bf6d5aec258cfdca"
server_key_version: "encryptionkey@0"
I20260429 08:01:18.844488 22227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "124e707237ff49239814996cef82e464"
format_stamp: "Formatted at 2026-04-29 08:01:18 on dist-test-slave-nhqz"
server_key: "396a60e02b8c20609b32274894953780"
server_key_iv: "a8952d18209a32d4bf6d5aec258cfdca"
server_key_version: "encryptionkey@0"
I20260429 08:01:18.847821 22227 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 08:01:18.850032 22241 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:18.850973 22227 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:01:18.851104 22227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "124e707237ff49239814996cef82e464"
format_stamp: "Formatted at 2026-04-29 08:01:18 on dist-test-slave-nhqz"
server_key: "396a60e02b8c20609b32274894953780"
server_key_iv: "a8952d18209a32d4bf6d5aec258cfdca"
server_key_version: "encryptionkey@0"
I20260429 08:01:18.851202 22227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:18.864367 22227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:18.865090 22227 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:18.865298 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:18.872218 22227 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:45901
I20260429 08:01:18.872228 22293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:45901 every 8 connection(s)
I20260429 08:01:18.873291 22227 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:01:18.875669 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22227
I20260429 08:01:18.875795 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:01:18.876073 12338 external_mini_cluster.cc:1468] Setting key 13404aca01a60a4ab1180d62bebf1daa
I20260429 08:01:18.876417 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:18.882997 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464: Bootstrap starting.
I20260429 08:01:18.883641 22297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:18.877362 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.12.12.190:45901)
0429 08:01:18.877898 (+ 536us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:18.877907 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:18.877939 (+ 32us) server_negotiation.cc:408] Connection header received
0429 08:01:18.878577 (+ 638us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:18.878595 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:18.878890 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:18.879137 (+ 247us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:18.879739 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.880528 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:18.881394 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:18.881596 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:18.882138 (+ 542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:18.882175 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:18.882185 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:18.882542 (+ 357us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:18.882576 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:18.882588 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:18.882661 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:18.882838 (+ 177us) server_negotiation.cc:300] Negotiation successful
0429 08:01:18.882993 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":148,"threads_started":1}
I20260429 08:01:18.885270 22294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:18.885927 22294 log.cc:826] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:18.887666 22294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464: No bootstrap required, opened a new log
I20260429 08:01:18.890055 22294 raft_consensus.cc:359] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e707237ff49239814996cef82e464" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45901 } }
I20260429 08:01:18.890303 22294 raft_consensus.cc:385] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:18.890401 22294 raft_consensus.cc:740] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124e707237ff49239814996cef82e464, State: Initialized, Role: FOLLOWER
I20260429 08:01:18.890802 22294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e707237ff49239814996cef82e464" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45901 } }
I20260429 08:01:18.890934 22294 raft_consensus.cc:399] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:01:18.890993 22294 raft_consensus.cc:493] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:01:18.891114 22294 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:18.891911 22294 raft_consensus.cc:515] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e707237ff49239814996cef82e464" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45901 } }
I20260429 08:01:18.892238 22294 leader_election.cc:304] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 124e707237ff49239814996cef82e464; no voters:
I20260429 08:01:18.892505 22294 leader_election.cc:290] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:01:18.892639 22299 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:01:18.892884 22299 raft_consensus.cc:697] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [term 1 LEADER]: Becoming Leader. State: Replica: 124e707237ff49239814996cef82e464, State: Running, Role: LEADER
I20260429 08:01:18.893164 22299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e707237ff49239814996cef82e464" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45901 } }
I20260429 08:01:18.893575 22294 sys_catalog.cc:565] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:01:18.894860 22301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124e707237ff49239814996cef82e464. Latest consensus state: current_term: 1 leader_uuid: "124e707237ff49239814996cef82e464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e707237ff49239814996cef82e464" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45901 } } }
I20260429 08:01:18.894840 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "124e707237ff49239814996cef82e464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e707237ff49239814996cef82e464" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 45901 } } }
I20260429 08:01:18.894992 22301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:18.895012 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464 [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:18.895478 22308 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:01:18.898259 22308 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:01:18.903321 22308 catalog_manager.cc:1357] Generated new cluster ID: bc22e28054044661aa5ba332892a9ba2
I20260429 08:01:18.903389 22308 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:01:18.916635 22308 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:01:18.917548 22308 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:01:18.927824 22308 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 124e707237ff49239814996cef82e464: Generated new TSK 0
I20260429 08:01:18.928409 22308 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:01:18.939604 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45901
--builtin_ntp_servers=127.12.12.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260429 08:01:19.041925 22318 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:19.042153 22318 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:19.042248 22318 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:19.045847 22318 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:19.045907 22318 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:19.046013 22318 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:01:19.050086 22318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:19.051571 22318 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:19.052739 22318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:19.059343 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.059408 22318 server_base.cc:1061] running on GCE node
W20260429 08:01:19.059362 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.059762 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.060187 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:19.060860 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:19.062067 22318 hybrid_clock.cc:648] HybridClock initialized: now 1777449679062044 us; error 41 us; skew 500 ppm
I20260429 08:01:19.064038 22318 webserver.cc:492] Webserver started at http://127.12.12.129:38023/ using document root <none> and password file <none>
I20260429 08:01:19.064666 22318 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:19.064754 22318 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:19.065001 22318 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:19.066772 22318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "25c4d6c123dd435db79c266d41354ac6"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "768b68ac52888265efdb196bf408cf48"
server_key_iv: "53ffc5e07276dc333ccb9cb37700fb6a"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.067265 22318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "25c4d6c123dd435db79c266d41354ac6"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "768b68ac52888265efdb196bf408cf48"
server_key_iv: "53ffc5e07276dc333ccb9cb37700fb6a"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.070559 22318 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 08:01:19.072790 22332 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.073881 22318 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:01:19.074018 22318 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "25c4d6c123dd435db79c266d41354ac6"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "768b68ac52888265efdb196bf408cf48"
server_key_iv: "53ffc5e07276dc333ccb9cb37700fb6a"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.074124 22318 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:19.093515 22318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:19.094173 22318 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:19.094379 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:19.094970 22318 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:19.096000 22318 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:19.096071 22318 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.096143 22318 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:19.096197 22318 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.104969 22318 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:36531
I20260429 08:01:19.104998 22445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:36531 every 8 connection(s)
I20260429 08:01:19.105944 22318 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:01:19.113976 22297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.107722 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:53483 (local address 127.12.12.190:45901)
0429 08:01:19.107876 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:19.107879 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:19.108678 (+ 799us) server_negotiation.cc:408] Connection header received
0429 08:01:19.109594 (+ 916us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:19.109597 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:19.109658 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:19.109763 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:19.110592 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.111079 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:19.111736 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.111915 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:19.113444 (+ 1529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:19.113463 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:19.113466 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:19.113479 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:19.113511 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:19.113520 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:19.113598 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:19.113770 (+ 172us) server_negotiation.cc:300] Negotiation successful
0429 08:01:19.113828 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 08:01:19.114571 22448 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.107977 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45901 (local address 127.12.12.129:53483)
0429 08:01:19.108533 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:19.108564 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:19.109383 (+ 819us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:19.109921 (+ 538us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:19.109929 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:19.109984 (+ 55us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:19.110449 (+ 465us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:19.110458 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.111201 (+ 743us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:19.111205 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:19.111596 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:19.111602 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.111853 (+ 251us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:19.113230 (+ 1377us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:01:19.113232 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:19.113263 (+ 31us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:01:19.113307 (+ 44us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:19.113614 (+ 307us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:19.113622 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:19.113634 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 08:01:19.113772 (+ 138us) client_negotiation.cc:241] Negotiation successful
0429 08:01:19.113910 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260429 08:01:19.115607 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22318
I20260429 08:01:19.115734 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:01:19.115756 22446 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45901
I20260429 08:01:19.115988 22446 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:19.116000 12338 external_mini_cluster.cc:1468] Setting key 5ca1428678a2a84fc5f13341de22e562
I20260429 08:01:19.116631 22446 heartbeater.cc:507] Master 127.12.12.190:45901 requested a full tablet report, sending...
I20260429 08:01:19.117959 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45901
--builtin_ntp_servers=127.12.12.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260429 08:01:19.118074 22258 ts_manager.cc:194] Registered new tserver with Master: 25c4d6c123dd435db79c266d41354ac6 (127.12.12.129:36531)
I20260429 08:01:19.119222 22258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.12.129:53483
W20260429 08:01:19.218079 22449 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:19.218299 22449 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:19.218361 22449 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:19.221592 22449 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:19.221692 22449 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:19.221763 22449 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:01:19.225626 22449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:19.226656 22449 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:19.227684 22449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:19.234118 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.234120 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.234123 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.234432 22449 server_base.cc:1061] running on GCE node
I20260429 08:01:19.234799 22449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:19.235368 22449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:19.236567 22449 hybrid_clock.cc:648] HybridClock initialized: now 1777449679236557 us; error 26 us; skew 500 ppm
I20260429 08:01:19.238435 22449 webserver.cc:492] Webserver started at http://127.12.12.130:39981/ using document root <none> and password file <none>
I20260429 08:01:19.239010 22449 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:19.239095 22449 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:19.239322 22449 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:19.241082 22449 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "d0ed3f379f0d4a0b95262788c62bd139"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "d9871e72a8d85c74be9a95666c7e516b"
server_key_iv: "7b358d524df367e1640b7ee823b59a99"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.241569 22449 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "d0ed3f379f0d4a0b95262788c62bd139"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "d9871e72a8d85c74be9a95666c7e516b"
server_key_iv: "7b358d524df367e1640b7ee823b59a99"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.244908 22449 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 08:01:19.247275 22463 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.248234 22449 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:19.248371 22449 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "d0ed3f379f0d4a0b95262788c62bd139"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "d9871e72a8d85c74be9a95666c7e516b"
server_key_iv: "7b358d524df367e1640b7ee823b59a99"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.248473 22449 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:19.260811 22449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:19.261488 22449 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:19.261667 22449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:19.262166 22449 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:19.263082 22449 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:19.263182 22449 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.263252 22449 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:19.263311 22449 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.272591 22449 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:45031
I20260429 08:01:19.272616 22576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:45031 every 8 connection(s)
I20260429 08:01:19.273532 22449 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:01:19.274011 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22449
I20260429 08:01:19.274132 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:01:19.274396 12338 external_mini_cluster.cc:1468] Setting key f3ad345882f2765e94b0bf4c46547b41
I20260429 08:01:19.276702 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45901
--builtin_ntp_servers=127.12.12.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260429 08:01:19.282440 22297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.275349 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:45109 (local address 127.12.12.190:45901)
0429 08:01:19.275483 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:19.275487 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:19.276612 (+ 1125us) server_negotiation.cc:408] Connection header received
0429 08:01:19.277841 (+ 1229us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:19.277844 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:19.277891 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:19.277968 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:19.278953 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.279747 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:19.280440 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.280697 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:19.281835 (+ 1138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:19.281861 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:19.281865 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:19.281881 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:19.281909 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:19.281922 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:19.282007 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:19.282163 (+ 156us) server_negotiation.cc:300] Negotiation successful
0429 08:01:19.282234 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 08:01:19.282893 22579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.275720 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45901 (local address 127.12.12.130:45109)
0429 08:01:19.276449 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:19.276494 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:19.277623 (+ 1129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:19.278186 (+ 563us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:19.278199 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:19.278289 (+ 90us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:19.278804 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:19.278815 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.279867 (+ 1052us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:19.279870 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:19.280320 (+ 450us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:19.280327 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.280486 (+ 159us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:19.281633 (+ 1147us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:01:19.281635 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:19.281659 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:01:19.281701 (+ 42us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:19.282022 (+ 321us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:19.282027 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:19.282031 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 08:01:19.282134 (+ 103us) client_negotiation.cc:241] Negotiation successful
0429 08:01:19.282249 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":124,"threads_started":1}
I20260429 08:01:19.283998 22577 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45901
I20260429 08:01:19.284224 22577 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:19.284761 22577 heartbeater.cc:507] Master 127.12.12.190:45901 requested a full tablet report, sending...
I20260429 08:01:19.285810 22258 ts_manager.cc:194] Registered new tserver with Master: d0ed3f379f0d4a0b95262788c62bd139 (127.12.12.130:45031)
I20260429 08:01:19.286341 22258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.12.130:45109
W20260429 08:01:19.379088 22580 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:19.379309 22580 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:19.379352 22580 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:19.382637 22580 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:19.382688 22580 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:19.382798 22580 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:01:19.386746 22580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:45901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:19.387899 22580 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:19.389281 22580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:19.395624 22588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.395697 22586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.395659 22585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.396071 22580 server_base.cc:1061] running on GCE node
I20260429 08:01:19.396453 22580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:19.397074 22580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:19.398222 22580 hybrid_clock.cc:648] HybridClock initialized: now 1777449679398205 us; error 28 us; skew 500 ppm
I20260429 08:01:19.399930 22580 webserver.cc:492] Webserver started at http://127.12.12.131:41875/ using document root <none> and password file <none>
I20260429 08:01:19.400516 22580 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:19.400568 22580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:19.400766 22580 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:19.402539 22580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "9c0de6f446be4f1391f937d8a150fd2e"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "b26c6e6768c902409229d8fb62709703"
server_key_iv: "f13bdeff5433be8603ed0a4df6babf0b"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.403077 22580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "9c0de6f446be4f1391f937d8a150fd2e"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "b26c6e6768c902409229d8fb62709703"
server_key_iv: "f13bdeff5433be8603ed0a4df6babf0b"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.406302 22580 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260429 08:01:19.408450 22594 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.409516 22580 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:01:19.409652 22580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "9c0de6f446be4f1391f937d8a150fd2e"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "b26c6e6768c902409229d8fb62709703"
server_key_iv: "f13bdeff5433be8603ed0a4df6babf0b"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.409767 22580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:19.428156 22580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:19.428910 22580 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:19.429105 22580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:19.429668 22580 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:19.430634 22580 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:19.430706 22580 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.430775 22580 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:19.430810 22580 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.439692 22580 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:41865
I20260429 08:01:19.439752 22707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:41865 every 8 connection(s)
I20260429 08:01:19.440625 22580 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:01:19.443692 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22580
I20260429 08:01:19.443785 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:01:19.444028 12338 external_mini_cluster.cc:1468] Setting key 9846444d42e3286ab803f2d1485abd29
I20260429 08:01:19.448495 22297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.442360 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:32911 (local address 127.12.12.190:45901)
0429 08:01:19.442526 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:19.442529 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:19.443363 (+ 834us) server_negotiation.cc:408] Connection header received
0429 08:01:19.444356 (+ 993us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:19.444361 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:19.444429 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:19.444537 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:19.445467 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.446057 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:19.446750 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.446896 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:19.448040 (+ 1144us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:19.448053 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:19.448055 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:19.448066 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:19.448085 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:19.448092 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:19.448151 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:19.448344 (+ 193us) server_negotiation.cc:300] Negotiation successful
0429 08:01:19.448390 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 08:01:19.449153 22710 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.442619 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:45901 (local address 127.12.12.131:32911)
0429 08:01:19.443220 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:19.443256 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:19.444102 (+ 846us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:19.444693 (+ 591us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:19.444701 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:19.444761 (+ 60us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:19.445301 (+ 540us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:19.445312 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.446230 (+ 918us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:19.446233 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:19.446614 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:19.446621 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:19.446806 (+ 185us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:19.447832 (+ 1026us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:01:19.447834 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:19.447858 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:01:19.447903 (+ 45us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:19.448169 (+ 266us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:19.448174 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:19.448178 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 08:01:19.448333 (+ 155us) client_negotiation.cc:241] Negotiation successful
0429 08:01:19.448444 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
I20260429 08:01:19.450176 22708 heartbeater.cc:344] Connected to a master server at 127.12.12.190:45901
I20260429 08:01:19.450429 22708 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:19.450930 22708 heartbeater.cc:507] Master 127.12.12.190:45901 requested a full tablet report, sending...
I20260429 08:01:19.451900 22258 ts_manager.cc:194] Registered new tserver with Master: 9c0de6f446be4f1391f937d8a150fd2e (127.12.12.131:41865)
I20260429 08:01:19.452513 22258 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.12.12.131:32911
I20260429 08:01:19.457790 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 08:01:19.460336 22297 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:19.459499 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.12.12.190:45901)
0429 08:01:19.459661 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:19.459664 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:19.459732 (+ 68us) server_negotiation.cc:408] Connection header received
0429 08:01:19.459827 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:19.459829 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:19.459869 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:19.459952 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:19.460213 (+ 261us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35778: BlockingRecv error: recv got EOF from 127.0.0.1:35778 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260429 08:01:19.460795 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22318
I20260429 08:01:19.467013 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22449
I20260429 08:01:19.472848 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22580
I20260429 08:01:19.478511 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22227
2026-04-29T08:01:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (786 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-04-29T08:01:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:01:19Z Disabled control of system clock
I20260429 08:01:19.507171 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:35565
--webserver_interface=127.12.12.190
--webserver_port=0
--builtin_ntp_servers=127.12.12.148:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:35565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 08:01:19.610608 22719 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:19.610810 22719 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:19.610857 22719 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:19.614311 22719 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:01:19.614400 22719 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:19.614418 22719 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:19.614436 22719 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:01:19.614466 22719 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:01:19.618326 22719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:35565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:35565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:19.619354 22719 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:19.620425 22719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:19.626168 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.626168 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.626117 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.626448 22719 server_base.cc:1061] running on GCE node
I20260429 08:01:19.626963 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:19.627897 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:19.629092 22719 hybrid_clock.cc:648] HybridClock initialized: now 1777449679629081 us; error 32 us; skew 500 ppm
I20260429 08:01:19.630833 22719 webserver.cc:492] Webserver started at http://127.12.12.190:39047/ using document root <none> and password file <none>
I20260429 08:01:19.631453 22719 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:19.631529 22719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:19.631721 22719 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:19.633437 22719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "8afdea2f05ea45ebbc3207366895d26f"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "ed37618845a04ec3f6aff2edec6acafb"
server_key_iv: "4bb4905cebff4c6f3acecd0e91380d7a"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.633903 22719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "8afdea2f05ea45ebbc3207366895d26f"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "ed37618845a04ec3f6aff2edec6acafb"
server_key_iv: "4bb4905cebff4c6f3acecd0e91380d7a"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.636986 22719 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:01:19.639046 22733 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.640002 22719 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:19.640131 22719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "8afdea2f05ea45ebbc3207366895d26f"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "ed37618845a04ec3f6aff2edec6acafb"
server_key_iv: "4bb4905cebff4c6f3acecd0e91380d7a"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.640229 22719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:19.644702 22719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:19.645262 22719 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:19.645430 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:19.652288 22719 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:35565
I20260429 08:01:19.652298 22785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:35565 every 8 connection(s)
I20260429 08:01:19.653332 22719 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:01:19.656101 22786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:19.661347 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f: Bootstrap starting.
I20260429 08:01:19.661986 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22719
I20260429 08:01:19.662120 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:01:19.662350 12338 external_mini_cluster.cc:1468] Setting key c71d4ba26f8a64e9dc85d8c7c640e0d1
I20260429 08:01:19.663930 22786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:19.664659 22786 log.cc:826] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:19.666589 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f: No bootstrap required, opened a new log
I20260429 08:01:19.667135 22789 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.663538 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.12.12.190:35565)
0429 08:01:19.664115 (+ 577us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:19.664123 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:19.664154 (+ 31us) server_negotiation.cc:408] Connection header received
0429 08:01:19.664657 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:19.664672 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:19.665010 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:19.665310 (+ 300us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:19.666144 (+ 834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:19.666149 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:19.666158 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:19.666167 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:19.666215 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:19.666225 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:19.666350 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:19.666484 (+ 134us) server_negotiation.cc:300] Negotiation successful
0429 08:01:19.666605 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260429 08:01:19.669153 22786 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afdea2f05ea45ebbc3207366895d26f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 35565 } }
I20260429 08:01:19.669337 22786 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:19.669435 22786 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8afdea2f05ea45ebbc3207366895d26f, State: Initialized, Role: FOLLOWER
I20260429 08:01:19.669827 22786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afdea2f05ea45ebbc3207366895d26f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 35565 } }
I20260429 08:01:19.669970 22786 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:01:19.670056 22786 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:01:19.670152 22786 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:19.671015 22786 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afdea2f05ea45ebbc3207366895d26f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 35565 } }
I20260429 08:01:19.671357 22786 leader_election.cc:304] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8afdea2f05ea45ebbc3207366895d26f; no voters:
I20260429 08:01:19.671662 22786 leader_election.cc:290] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:01:19.671926 22791 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:01:19.672127 22791 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [term 1 LEADER]: Becoming Leader. State: Replica: 8afdea2f05ea45ebbc3207366895d26f, State: Running, Role: LEADER
I20260429 08:01:19.672406 22791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afdea2f05ea45ebbc3207366895d26f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 35565 } }
I20260429 08:01:19.672783 22786 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:01:19.673972 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8afdea2f05ea45ebbc3207366895d26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afdea2f05ea45ebbc3207366895d26f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 35565 } } }
I20260429 08:01:19.674080 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:19.674937 22800 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:01:19.675050 22793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8afdea2f05ea45ebbc3207366895d26f. Latest consensus state: current_term: 1 leader_uuid: "8afdea2f05ea45ebbc3207366895d26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8afdea2f05ea45ebbc3207366895d26f" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 35565 } } }
I20260429 08:01:19.675190 22793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:19.678007 22800 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:01:19.682689 22800 catalog_manager.cc:1357] Generated new cluster ID: 8c8e41e12bf34f0fa2ef2382ff9ca876
I20260429 08:01:19.682754 22800 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:01:19.692346 22800 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:01:19.692682 22800 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:01:19.703708 22800 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 8afdea2f05ea45ebbc3207366895d26f: Generated new TSK 0
I20260429 08:01:19.704373 22800 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 08:01:19.714105 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:35565
--builtin_ntp_servers=127.12.12.148:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 08:01:19.813946 22810 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:19.814142 22810 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:19.814225 22810 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:19.817510 22810 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:19.817565 22810 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:19.817679 22810 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:01:19.821591 22810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:19.822639 22810 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:19.823755 22810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:19.830013 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.830024 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.830235 22810 server_base.cc:1061] running on GCE node
W20260429 08:01:19.830045 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.830766 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:19.831346 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:19.832540 22810 hybrid_clock.cc:648] HybridClock initialized: now 1777449679832519 us; error 49 us; skew 500 ppm
I20260429 08:01:19.834295 22810 webserver.cc:492] Webserver started at http://127.12.12.129:42369/ using document root <none> and password file <none>
I20260429 08:01:19.834831 22810 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:19.834913 22810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:19.835127 22810 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:19.836740 22810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "bf2a2d0a941f4c8692fcaffc4d747533"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "89e1c8da0ae7db601f9d40f4aafc0f2b"
server_key_iv: "2354f46151feb5f6b91a665a5e3061f9"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.837265 22810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2a2d0a941f4c8692fcaffc4d747533"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "89e1c8da0ae7db601f9d40f4aafc0f2b"
server_key_iv: "2354f46151feb5f6b91a665a5e3061f9"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.840322 22810 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 08:01:19.842579 22824 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.843510 22810 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:01:19.843638 22810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "bf2a2d0a941f4c8692fcaffc4d747533"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "89e1c8da0ae7db601f9d40f4aafc0f2b"
server_key_iv: "2354f46151feb5f6b91a665a5e3061f9"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.843757 22810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:19.848201 22810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:19.848708 22810 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:19.848902 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:19.849423 22810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:19.850287 22810 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:19.850359 22810 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.850425 22810 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:19.850477 22810 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:19.858768 22810 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:33077
I20260429 08:01:19.858839 22937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:33077 every 8 connection(s)
I20260429 08:01:19.859668 22810 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:01:19.866115 22789 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.862933 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:38815 (local address 127.12.12.190:35565)
0429 08:01:19.863103 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:19.863110 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:19.863131 (+ 21us) server_negotiation.cc:408] Connection header received
0429 08:01:19.864011 (+ 880us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:19.864017 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:19.864073 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:19.864171 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:19.865511 (+ 1340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:19.865519 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:19.865523 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:19.865539 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:19.865561 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:19.865570 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:19.865674 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:19.865864 (+ 190us) server_negotiation.cc:300] Negotiation successful
0429 08:01:19.865932 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 08:01:19.866757 22940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:19.861927 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:35565 (local address 127.12.12.129:38815)
0429 08:01:19.862677 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:19.862723 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:19.863754 (+ 1031us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:19.864349 (+ 595us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:19.864359 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:19.864457 (+ 98us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:19.865190 (+ 733us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:01:19.865193 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:19.865231 (+ 38us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:01:19.865292 (+ 61us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:19.865679 (+ 387us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:19.865688 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:19.865694 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 08:01:19.865883 (+ 189us) client_negotiation.cc:241] Negotiation successful
0429 08:01:19.865988 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
I20260429 08:01:19.868229 22938 heartbeater.cc:344] Connected to a master server at 127.12.12.190:35565
I20260429 08:01:19.868513 22938 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:19.869063 22938 heartbeater.cc:507] Master 127.12.12.190:35565 requested a full tablet report, sending...
I20260429 08:01:19.869580 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22810
I20260429 08:01:19.869716 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:01:19.870065 12338 external_mini_cluster.cc:1468] Setting key a3cbe2f020cdf14a35b76ade80d62501
I20260429 08:01:19.870735 22750 ts_manager.cc:194] Registered new tserver with Master: bf2a2d0a941f4c8692fcaffc4d747533 (127.12.12.129:33077)
I20260429 08:01:19.872386 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:35565
--builtin_ntp_servers=127.12.12.148:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 08:01:19.973138 22941 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:19.973359 22941 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:19.973409 22941 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:19.976783 22941 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:19.976902 22941 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:19.977017 22941 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:01:19.980836 22941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:19.981930 22941 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:19.983031 22941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:19.989614 22947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.989604 22946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:19.989604 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:19.989862 22941 server_base.cc:1061] running on GCE node
I20260429 08:01:19.990254 22941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:19.990820 22941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:19.991997 22941 hybrid_clock.cc:648] HybridClock initialized: now 1777449679991974 us; error 34 us; skew 500 ppm
I20260429 08:01:19.993938 22941 webserver.cc:492] Webserver started at http://127.12.12.130:41587/ using document root <none> and password file <none>
I20260429 08:01:19.994565 22941 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:19.994621 22941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:19.994858 22941 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:19.996552 22941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "52d3449b07fa4baeae04c915f1849fad"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "e1ad43f83d83f59d10bab33c90b002f7"
server_key_iv: "678bb796825f677a10f045fbe76cdbb7"
server_key_version: "encryptionkey@0"
I20260429 08:01:19.997066 22941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "52d3449b07fa4baeae04c915f1849fad"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "e1ad43f83d83f59d10bab33c90b002f7"
server_key_iv: "678bb796825f677a10f045fbe76cdbb7"
server_key_version: "encryptionkey@0"
I20260429 08:01:20.000450 22941 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260429 08:01:20.002714 22955 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:20.003865 22941 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260429 08:01:20.004021 22941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "52d3449b07fa4baeae04c915f1849fad"
format_stamp: "Formatted at 2026-04-29 08:01:19 on dist-test-slave-nhqz"
server_key: "e1ad43f83d83f59d10bab33c90b002f7"
server_key_iv: "678bb796825f677a10f045fbe76cdbb7"
server_key_version: "encryptionkey@0"
I20260429 08:01:20.004134 22941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:20.008895 22941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:20.009473 22941 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:20.009663 22941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:20.010176 22941 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:20.011125 22941 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:20.011197 22941 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:20.011271 22941 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:20.011315 22941 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:20.020360 22941 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:46095
I20260429 08:01:20.020390 23068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:46095 every 8 connection(s)
I20260429 08:01:20.021378 22941 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:01:20.026495 22789 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:20.023125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:46047 (local address 127.12.12.190:35565)
0429 08:01:20.023305 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:20.023310 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:20.024102 (+ 792us) server_negotiation.cc:408] Connection header received
0429 08:01:20.024900 (+ 798us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:20.024903 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:20.024945 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:20.025022 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:20.026109 (+ 1087us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:20.026112 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:20.026114 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:20.026124 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:20.026143 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:20.026150 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:20.026228 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:20.026351 (+ 123us) server_negotiation.cc:300] Negotiation successful
0429 08:01:20.026381 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260429 08:01:20.027205 23071 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:20.023417 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:35565 (local address 127.12.12.130:46047)
0429 08:01:20.023969 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:20.024001 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:20.024676 (+ 675us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:20.025169 (+ 493us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:20.025205 (+ 36us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:20.025289 (+ 84us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:20.025903 (+ 614us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:01:20.025905 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:20.025929 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:01:20.025976 (+ 47us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:20.026224 (+ 248us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:20.026228 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:20.026232 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 08:01:20.026370 (+ 138us) client_negotiation.cc:241] Negotiation successful
0429 08:01:20.026468 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260429 08:01:20.027901 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 22941
I20260429 08:01:20.028051 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:01:20.028323 12338 external_mini_cluster.cc:1468] Setting key cb8769d217a9dfb73a909916ba9a28dd
I20260429 08:01:20.028730 23069 heartbeater.cc:344] Connected to a master server at 127.12.12.190:35565
I20260429 08:01:20.028997 23069 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:20.029381 23069 heartbeater.cc:507] Master 127.12.12.190:35565 requested a full tablet report, sending...
I20260429 08:01:20.030227 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:35565
--builtin_ntp_servers=127.12.12.148:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260429 08:01:20.030423 22750 ts_manager.cc:194] Registered new tserver with Master: 52d3449b07fa4baeae04c915f1849fad (127.12.12.130:46095)
W20260429 08:01:20.132656 23072 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:20.133111 23072 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:20.133272 23072 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:20.136978 23072 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:20.137034 23072 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:20.137166 23072 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:01:20.141106 23072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:20.142144 23072 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:20.143271 23072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 08:01:20.149791 23072 server_base.cc:1061] running on GCE node
W20260429 08:01:20.149740 23080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:20.149734 23077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:20.149768 23078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:20.150297 23072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:20.150863 23072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:20.152037 23072 hybrid_clock.cc:648] HybridClock initialized: now 1777449680152017 us; error 31 us; skew 500 ppm
I20260429 08:01:20.154003 23072 webserver.cc:492] Webserver started at http://127.12.12.131:44831/ using document root <none> and password file <none>
I20260429 08:01:20.154578 23072 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:20.154634 23072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:20.154879 23072 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:20.156492 23072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "007569f0e25a4c5f8881186ab7167009"
format_stamp: "Formatted at 2026-04-29 08:01:20 on dist-test-slave-nhqz"
server_key: "da3db3dd6900bc68621ed13271612b96"
server_key_iv: "2d309718a7c200f078f3d0611e151021"
server_key_version: "encryptionkey@0"
I20260429 08:01:20.157018 23072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "007569f0e25a4c5f8881186ab7167009"
format_stamp: "Formatted at 2026-04-29 08:01:20 on dist-test-slave-nhqz"
server_key: "da3db3dd6900bc68621ed13271612b96"
server_key_iv: "2d309718a7c200f078f3d0611e151021"
server_key_version: "encryptionkey@0"
I20260429 08:01:20.160164 23072 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260429 08:01:20.162456 23086 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:20.163496 23072 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 08:01:20.163640 23072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "007569f0e25a4c5f8881186ab7167009"
format_stamp: "Formatted at 2026-04-29 08:01:20 on dist-test-slave-nhqz"
server_key: "da3db3dd6900bc68621ed13271612b96"
server_key_iv: "2d309718a7c200f078f3d0611e151021"
server_key_version: "encryptionkey@0"
I20260429 08:01:20.163748 23072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:20.168404 23072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:20.169034 23072 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:20.169214 23072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:20.169867 23072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:20.170845 23072 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:20.170919 23072 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:20.170987 23072 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:20.171029 23072 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:20.180483 23072 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:46765
I20260429 08:01:20.180546 23199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:46765 every 8 connection(s)
I20260429 08:01:20.181591 23072 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:01:20.185979 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 23072
I20260429 08:01:20.186111 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:01:20.186357 12338 external_mini_cluster.cc:1468] Setting key f01799f7432a96424834fb185b4b01bc
I20260429 08:01:20.186877 22789 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:20.183317 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:37401 (local address 127.12.12.190:35565)
0429 08:01:20.183463 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:20.183466 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:20.184312 (+ 846us) server_negotiation.cc:408] Connection header received
0429 08:01:20.185186 (+ 874us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:20.185189 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:20.185236 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:20.185355 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:20.186403 (+ 1048us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:20.186407 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:20.186416 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 08:01:20.186427 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:20.186446 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 08:01:20.186453 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:20.186554 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:20.186702 (+ 148us) server_negotiation.cc:300] Negotiation successful
0429 08:01:20.186740 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:01:20.187455 23202 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:20.183631 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:35565 (local address 127.12.12.131:37401)
0429 08:01:20.184172 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:20.184204 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:20.184988 (+ 784us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:20.185513 (+ 525us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:20.185522 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:20.185584 (+ 62us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:20.186192 (+ 608us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 08:01:20.186194 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:20.186216 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 08:01:20.186261 (+ 45us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:20.186540 (+ 279us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:20.186546 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:20.186551 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 08:01:20.186750 (+ 199us) client_negotiation.cc:241] Negotiation successful
0429 08:01:20.186853 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260429 08:01:20.188889 23200 heartbeater.cc:344] Connected to a master server at 127.12.12.190:35565
I20260429 08:01:20.189121 23200 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:20.189533 23200 heartbeater.cc:507] Master 127.12.12.190:35565 requested a full tablet report, sending...
I20260429 08:01:20.190449 22750 ts_manager.cc:194] Registered new tserver with Master: 007569f0e25a4c5f8881186ab7167009 (127.12.12.131:46765)
I20260429 08:01:20.200548 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 08:01:20.203408 22789 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:01:20.202468 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.12.12.190:35565)
0429 08:01:20.202631 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:20.202635 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:20.202766 (+ 131us) server_negotiation.cc:408] Connection header received
0429 08:01:20.202918 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:20.202921 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:20.202965 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:20.203075 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:20.203290 (+ 215us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39310: BlockingRecv error: recv got EOF from 127.0.0.1:39310 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260429 08:01:20.203787 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22810
I20260429 08:01:20.209750 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22941
I20260429 08:01:20.215251 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 23072
I20260429 08:01:20.220726 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 22719
2026-04-29T08:01:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (742 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:01:20 dist-test-slave-nhqz krb5kdc[23210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:01:20 dist-test-slave-nhqz krb5kdc[23210](info): set up 2 sockets
Apr 29 08:01:20 dist-test-slave-nhqz krb5kdc[23210](info): commencing operation
krb5kdc: starting...
W20260429 08:01:22.210527 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:01:22 dist-test-slave-nhqz krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:01:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:22.348343 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34995
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:22.455776 23226 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:22.456055 23226 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:22.456143 23226 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:22.459497 23226 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:01:22.459601 23226 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:22.459648 23226 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:22.459709 23226 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:01:22.459751 23226 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:01:22.464303 23226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:34995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:22.465413 23226 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:22.466293 23226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:22.472061 23234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:22.472047 23232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:22.472127 23226 server_base.cc:1061] running on GCE node
W20260429 08:01:22.472059 23231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:22.472749 23226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:22.473770 23226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:22.474951 23226 hybrid_clock.cc:648] HybridClock initialized: now 1777449682474936 us; error 36 us; skew 500 ppm
Apr 29 08:01:22 dist-test-slave-nhqz krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:22.477885 23226 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:01:22.478994 23226 webserver.cc:492] Webserver started at http://127.12.12.190:36179/ using document root <none> and password file <none>
I20260429 08:01:22.479568 23226 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:22.479643 23226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:22.479847 23226 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:22.481582 23226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "159f05743a814237a443a61fdd0768e1"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "f8eb6d3060d0fbb32a842a26954a7fe6"
server_key_iv: "dc24e5e9375654e8a534fd830a4a6ef6"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.482071 23226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "159f05743a814237a443a61fdd0768e1"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "f8eb6d3060d0fbb32a842a26954a7fe6"
server_key_iv: "dc24e5e9375654e8a534fd830a4a6ef6"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.485553 23226 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:01:22.487731 23241 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:22.488924 23226 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 08:01:22.489054 23226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "159f05743a814237a443a61fdd0768e1"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "f8eb6d3060d0fbb32a842a26954a7fe6"
server_key_iv: "dc24e5e9375654e8a534fd830a4a6ef6"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.489161 23226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:22.512833 23226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:22.517040 23226 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:22.517256 23226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:22.526299 23226 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:34995
I20260429 08:01:22.527801 23226 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:01:22.526301 23293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:34995 every 8 connection(s)
I20260429 08:01:22.531571 23294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:22.535667 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 23226
I20260429 08:01:22.535813 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:01:22.536125 12338 external_mini_cluster.cc:1468] Setting key d2c1471a4afad19900ae000cbf6055cc
I20260429 08:01:22.539259 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1: Bootstrap starting.
I20260429 08:01:22.542120 23294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:22.543016 23294 log.cc:826] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:22.545202 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1: No bootstrap required, opened a new log
Apr 29 08:01:22 dist-test-slave-nhqz krb5kdc[23210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:01:22.548862 23294 raft_consensus.cc:359] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159f05743a814237a443a61fdd0768e1" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34995 } }
I20260429 08:01:22.549109 23294 raft_consensus.cc:385] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:22.549196 23294 raft_consensus.cc:740] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159f05743a814237a443a61fdd0768e1, State: Initialized, Role: FOLLOWER
I20260429 08:01:22.549736 23294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159f05743a814237a443a61fdd0768e1" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34995 } }
I20260429 08:01:22.549888 23294 raft_consensus.cc:399] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:01:22.549963 23294 raft_consensus.cc:493] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:01:22.550078 23294 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:22.551224 23294 raft_consensus.cc:515] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159f05743a814237a443a61fdd0768e1" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34995 } }
I20260429 08:01:22.551648 23294 leader_election.cc:304] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 159f05743a814237a443a61fdd0768e1; no voters:
I20260429 08:01:22.551967 23294 leader_election.cc:290] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:01:22.552313 23299 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:01:22.552560 23299 raft_consensus.cc:697] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [term 1 LEADER]: Becoming Leader. State: Replica: 159f05743a814237a443a61fdd0768e1, State: Running, Role: LEADER
I20260429 08:01:22.552933 23299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159f05743a814237a443a61fdd0768e1" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34995 } }
I20260429 08:01:22.553339 23294 sys_catalog.cc:565] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:01:22.553553 23297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:22.537562 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.12.12.190:34995)
0429 08:01:22.538104 (+ 542us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:22.538120 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:22.538165 (+ 45us) server_negotiation.cc:408] Connection header received
0429 08:01:22.538990 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:22.539018 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:22.539425 (+ 407us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:22.539813 (+ 388us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:22.541518 (+ 1705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:22.542856 (+ 1338us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:22.544171 (+ 1315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:22.544519 (+ 348us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:22.547243 (+ 2724us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:22.547270 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:22.547289 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:22.547332 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:22.549786 (+ 2454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:22.550377 (+ 591us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:22.550384 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:22.550390 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:22.550511 (+ 121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:22.550823 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:22.550828 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:22.550831 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:22.551318 (+ 487us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:22.551542 (+ 224us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:22.552363 (+ 821us) server_negotiation.cc:300] Negotiation successful
0429 08:01:22.552626 (+ 263us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":135,"threads_started":1}
I20260429 08:01:22.554581 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "159f05743a814237a443a61fdd0768e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159f05743a814237a443a61fdd0768e1" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34995 } } }
I20260429 08:01:22.554706 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:22.554991 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 159f05743a814237a443a61fdd0768e1. Latest consensus state: current_term: 1 leader_uuid: "159f05743a814237a443a61fdd0768e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159f05743a814237a443a61fdd0768e1" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 34995 } } }
I20260429 08:01:22.555061 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1 [sys.catalog]: This master's current role is: LEADER
I20260429 08:01:22.555823 23308 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:01:22.558761 23308 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:01:22.563834 23308 catalog_manager.cc:1357] Generated new cluster ID: e265334262e048bf9214d48d93662d57
I20260429 08:01:22.563956 23308 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:01:22.584306 23308 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:01:22.585511 23308 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:01:22.595897 23308 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1: Generated new TSK 0
I20260429 08:01:22.596623 23308 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:22.651969 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34995
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:22.754899 23322 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:22.755168 23322 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:22.755270 23322 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:22.758845 23322 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:22.758950 23322 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:22.759068 23322 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:01:22.763417 23322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:22.764559 23322 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:22.765501 23322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:22.771987 23327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:22.771987 23330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:22.772058 23328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:22.772219 23322 server_base.cc:1061] running on GCE node
I20260429 08:01:22.772557 23322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:22.773130 23322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:22.774310 23322 hybrid_clock.cc:648] HybridClock initialized: now 1777449682774300 us; error 34 us; skew 500 ppm
Apr 29 08:01:22 dist-test-slave-nhqz krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:22.777086 23322 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:01:22.778096 23322 webserver.cc:492] Webserver started at http://127.12.12.129:45781/ using document root <none> and password file <none>
I20260429 08:01:22.778614 23322 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:22.778658 23322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:22.778806 23322 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:22.780405 23322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "7456b22a74b767855c7c06dedabf5460"
server_key_iv: "3cbd77b9f01c7f1b300e8ee833da2ffe"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.780910 23322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "7456b22a74b767855c7c06dedabf5460"
server_key_iv: "3cbd77b9f01c7f1b300e8ee833da2ffe"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.784590 23322 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260429 08:01:22.786675 23337 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:22.787632 23322 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260429 08:01:22.787755 23322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "7456b22a74b767855c7c06dedabf5460"
server_key_iv: "3cbd77b9f01c7f1b300e8ee833da2ffe"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.787855 23322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:22.802685 23322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:22.805676 23322 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:22.805887 23322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:22.806460 23322 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:22.807340 23322 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:22.807387 23322 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:22.807474 23322 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:22.807512 23322 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:22.816461 23322 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:46021
I20260429 08:01:22.816538 23450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:46021 every 8 connection(s)
I20260429 08:01:22.817570 23322 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:01:22.818118 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 23322
I20260429 08:01:22.818271 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:01:22.818601 12338 external_mini_cluster.cc:1468] Setting key 5e7c98005e9d4daf76562cf4f0957e4a
Apr 29 08:01:22 dist-test-slave-nhqz krb5kdc[23210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
I20260429 08:01:22.830703 23297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:22.819537 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:40751 (local address 127.12.12.190:34995)
0429 08:01:22.819697 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:22.819702 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:22.820617 (+ 915us) server_negotiation.cc:408] Connection header received
0429 08:01:22.821578 (+ 961us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:22.821582 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:22.821641 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:22.821723 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:22.822882 (+ 1159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:22.823434 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:22.824078 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:22.824267 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:22.827057 (+ 2790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:22.827084 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:22.827088 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:22.827122 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:22.828527 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:22.829128 (+ 601us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:22.829131 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:22.829132 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:22.829176 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:22.829544 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:22.829547 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:22.829549 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:22.829754 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:22.829861 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:22.830469 (+ 608us) server_negotiation.cc:300] Negotiation successful
0429 08:01:22.830581 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260429 08:01:22.831591 23453 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:22.819922 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34995 (local address 127.12.12.129:40751)
0429 08:01:22.820426 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:22.820464 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:22.821366 (+ 902us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:22.821876 (+ 510us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:22.821885 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:22.822275 (+ 390us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:22.822739 (+ 464us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:22.822750 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:22.823575 (+ 825us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:22.823579 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:22.823962 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:22.823967 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:22.824181 (+ 214us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:22.825418 (+ 1237us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:22.825437 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:22.826891 (+ 1454us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:22.828640 (+ 1749us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:22.828645 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:22.828655 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:22.829035 (+ 380us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:22.829264 (+ 229us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:22.829266 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:22.829268 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:22.829384 (+ 116us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:22.829856 (+ 472us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:22.829861 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:22.830173 (+ 312us) client_negotiation.cc:770] Sending connection context
0429 08:01:22.830421 (+ 248us) client_negotiation.cc:241] Negotiation successful
0429 08:01:22.830679 (+ 258us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":131,"threads_started":1}
I20260429 08:01:22.832881 23451 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34995
I20260429 08:01:22.833149 23451 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:22.833796 23451 heartbeater.cc:507] Master 127.12.12.190:34995 requested a full tablet report, sending...
I20260429 08:01:22.835424 23251 ts_manager.cc:194] Registered new tserver with Master: 8b7729d3c6f84c48a3ddfc29ebfd1dc1 (127.12.12.129:46021)
I20260429 08:01:22.836599 23251 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:40751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:22.867250 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34995
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:22.972765 23458 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:22.973040 23458 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:22.973138 23458 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:22.976308 23458 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:22.976366 23458 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:22.976442 23458 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:01:22.980602 23458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:22.981613 23458 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:22.982470 23458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:22.988843 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:22.988878 23458 server_base.cc:1061] running on GCE node
W20260429 08:01:22.988822 23466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:22.988895 23463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:22.989420 23458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:22.989964 23458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:22.991142 23458 hybrid_clock.cc:648] HybridClock initialized: now 1777449682991131 us; error 23 us; skew 500 ppm
Apr 29 08:01:22 dist-test-slave-nhqz krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:22.993989 23458 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:01:22.994953 23458 webserver.cc:492] Webserver started at http://127.12.12.130:36871/ using document root <none> and password file <none>
I20260429 08:01:22.995474 23458 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:22.995523 23458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:22.995754 23458 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:22.997509 23458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "758fc064c26e436bafbde672e69717c8"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "a0af4e2842f3db22d4c8be6fbc0a548d"
server_key_iv: "b7fd2749c526b7dce0ae59e4a6b730b9"
server_key_version: "encryptionkey@0"
I20260429 08:01:22.997961 23458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "758fc064c26e436bafbde672e69717c8"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "a0af4e2842f3db22d4c8be6fbc0a548d"
server_key_iv: "b7fd2749c526b7dce0ae59e4a6b730b9"
server_key_version: "encryptionkey@0"
I20260429 08:01:23.001019 23458 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 08:01:23.003060 23473 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:23.004279 23458 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 08:01:23.004398 23458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "758fc064c26e436bafbde672e69717c8"
format_stamp: "Formatted at 2026-04-29 08:01:22 on dist-test-slave-nhqz"
server_key: "a0af4e2842f3db22d4c8be6fbc0a548d"
server_key_iv: "b7fd2749c526b7dce0ae59e4a6b730b9"
server_key_version: "encryptionkey@0"
I20260429 08:01:23.004495 23458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:23.014389 23458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:23.017086 23458 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:23.017292 23458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:23.017830 23458 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:23.018733 23458 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:23.018803 23458 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:23.018872 23458 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:23.018923 23458 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:23.028299 23458 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:46793
I20260429 08:01:23.028332 23586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:46793 every 8 connection(s)
I20260429 08:01:23.029345 23458 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:01:23.033419 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 23458
I20260429 08:01:23.033571 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:01:23.033824 12338 external_mini_cluster.cc:1468] Setting key 8a85640268d9f108fee2944596207ea7
Apr 29 08:01:23 dist-test-slave-nhqz krb5kdc[23210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449682, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:01:23.041796 23297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.031059 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:40851 (local address 127.12.12.190:34995)
0429 08:01:23.031217 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.031221 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.031834 (+ 613us) server_negotiation.cc:408] Connection header received
0429 08:01:23.032588 (+ 754us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.032591 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.032642 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.032716 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:23.034488 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.035070 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.035711 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.035933 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.037970 (+ 2037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:23.037992 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:23.037998 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:23.038032 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:23.039879 (+ 1847us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:23.040356 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:23.040363 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:23.040366 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:23.040424 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:23.040707 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:23.040711 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:23.040713 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:23.040926 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:23.041044 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.041488 (+ 444us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.041612 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 08:01:23.042300 23589 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.031316 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34995 (local address 127.12.12.130:40851)
0429 08:01:23.031700 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.031732 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.032411 (+ 679us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.032911 (+ 500us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.032919 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.033301 (+ 382us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:23.034347 (+ 1046us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.034360 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.035186 (+ 826us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.035190 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.035593 (+ 403us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.035600 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.035741 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.036294 (+ 553us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:23.036311 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:23.037813 (+ 1502us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:23.040002 (+ 2189us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:23.040007 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:23.040015 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:23.040237 (+ 222us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:23.040527 (+ 290us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:23.040530 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:23.040531 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:23.040609 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:23.041037 (+ 428us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:23.041041 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:23.041228 (+ 187us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.041401 (+ 173us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.041576 (+ 175us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:01:23.043368 23587 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34995
I20260429 08:01:23.043617 23587 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:23.044121 23587 heartbeater.cc:507] Master 127.12.12.190:34995 requested a full tablet report, sending...
I20260429 08:01:23.045176 23251 ts_manager.cc:194] Registered new tserver with Master: 758fc064c26e436bafbde672e69717c8 (127.12.12.130:46793)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
I20260429 08:01:23.045950 23251 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:40851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:01:23.079146 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:34995
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:01:23.182528 23594 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:01:23.182754 23594 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:01:23.182809 23594 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:01:23.186282 23594 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:01:23.186348 23594 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:01:23.186488 23594 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:01:23.190621 23594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:01:23.191609 23594 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:01:23.192409 23594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:01:23.198467 23600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:23.198487 23602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:01:23.198454 23599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:01:23.198819 23594 server_base.cc:1061] running on GCE node
I20260429 08:01:23.199154 23594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:01:23.199626 23594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:01:23.200778 23594 hybrid_clock.cc:648] HybridClock initialized: now 1777449683200755 us; error 36 us; skew 500 ppm
Apr 29 08:01:23 dist-test-slave-nhqz krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449683, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:01:23.203464 23594 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:01:23.204509 23594 webserver.cc:492] Webserver started at http://127.12.12.131:44685/ using document root <none> and password file <none>
I20260429 08:01:23.205171 23594 fs_manager.cc:362] Metadata directory not provided
I20260429 08:01:23.205256 23594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:01:23.205432 23594 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 08:01:23.207046 23594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "1372b069f81a46b9b07cd83503f5928e"
format_stamp: "Formatted at 2026-04-29 08:01:23 on dist-test-slave-nhqz"
server_key: "9bd3ae102c7dcf02c2da957ffd3e17fa"
server_key_iv: "65783672079536cdfdca85be6f160986"
server_key_version: "encryptionkey@0"
I20260429 08:01:23.207535 23594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "1372b069f81a46b9b07cd83503f5928e"
format_stamp: "Formatted at 2026-04-29 08:01:23 on dist-test-slave-nhqz"
server_key: "9bd3ae102c7dcf02c2da957ffd3e17fa"
server_key_iv: "65783672079536cdfdca85be6f160986"
server_key_version: "encryptionkey@0"
I20260429 08:01:23.210763 23594 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260429 08:01:23.212982 23609 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:23.213891 23594 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 08:01:23.214033 23594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "1372b069f81a46b9b07cd83503f5928e"
format_stamp: "Formatted at 2026-04-29 08:01:23 on dist-test-slave-nhqz"
server_key: "9bd3ae102c7dcf02c2da957ffd3e17fa"
server_key_iv: "65783672079536cdfdca85be6f160986"
server_key_version: "encryptionkey@0"
I20260429 08:01:23.214135 23594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:01:23.222951 23594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:01:23.225519 23594 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:01:23.225726 23594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:01:23.226263 23594 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:01:23.227113 23594 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:01:23.227186 23594 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:23.227253 23594 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:01:23.227305 23594 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:01:23.236479 23594 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:36425
I20260429 08:01:23.236503 23722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:36425 every 8 connection(s)
I20260429 08:01:23.237417 23594 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:01:23.244596 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 23594
I20260429 08:01:23.244711 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:01:23.245066 12338 external_mini_cluster.cc:1468] Setting key b1f9843a0657e528e8f0bf55d7143dd0
Apr 29 08:01:23 dist-test-slave-nhqz krb5kdc[23210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449683, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:01:23.249820 23297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.239155 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:50209 (local address 127.12.12.190:34995)
0429 08:01:23.239299 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.239303 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.239897 (+ 594us) server_negotiation.cc:408] Connection header received
0429 08:01:23.240712 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.240716 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.240776 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.240914 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:23.242042 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.242545 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.243219 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.243408 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.246405 (+ 2997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:23.246422 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:23.246425 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:23.246454 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:23.248115 (+ 1661us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:23.248548 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:23.248551 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:23.248552 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:23.248590 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:23.248869 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:23.248872 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:23.248873 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:23.249003 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:23.249084 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.249587 (+ 503us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.249689 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:01:23.250402 23725 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.239349 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:34995 (local address 127.12.12.131:50209)
0429 08:01:23.239758 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.239788 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.240515 (+ 727us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.241074 (+ 559us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.241082 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.241463 (+ 381us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:01:23.241888 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.241899 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.242695 (+ 796us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.242698 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.243097 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.243103 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.243315 (+ 212us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.244361 (+ 1046us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:01:23.244379 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:01:23.246181 (+ 1802us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:01:23.248216 (+ 2035us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:23.248221 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:23.248229 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:23.248460 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:23.248666 (+ 206us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:01:23.248668 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:01:23.248669 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:01:23.248746 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:01:23.249090 (+ 344us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:01:23.249094 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:01:23.249326 (+ 232us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.249487 (+ 161us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.249670 (+ 183us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":89,"threads_started":1}
I20260429 08:01:23.251539 23723 heartbeater.cc:344] Connected to a master server at 127.12.12.190:34995
I20260429 08:01:23.251766 23723 heartbeater.cc:461] Registering TS with master...
I20260429 08:01:23.252226 23723 heartbeater.cc:507] Master 127.12.12.190:34995 requested a full tablet report, sending...
I20260429 08:01:23.253286 23251 ts_manager.cc:194] Registered new tserver with Master: 1372b069f81a46b9b07cd83503f5928e (127.12.12.131:36425)
I20260429 08:01:23.253837 23251 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:50209
I20260429 08:01:23.258777 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 08:01:23.266815 23297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.260572 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.12.12.190:34995)
0429 08:01:23.260740 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.260743 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.260847 (+ 104us) server_negotiation.cc:408] Connection header received
0429 08:01:23.260970 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.260973 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.261011 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.261077 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:01:23.261841 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.262298 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.262890 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.263011 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.264039 (+ 1028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:01:23.264052 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:01:23.264055 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:01:23.264077 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:01:23.265511 (+ 1434us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:23.265847 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:23.265849 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:23.265851 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:23.265889 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:01:23.266069 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:01:23.266071 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:01:23.266072 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:01:23.266222 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:01:23.266321 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.266499 (+ 178us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.266586 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260429 08:01:23.269426 23251 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:01:23.271759 23251 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 08:01:23.284292 23734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.279829 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:46021 (local address 127.0.0.1:33068)
0429 08:01:23.280680 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.280714 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.280887 (+ 173us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.281482 (+ 595us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.281487 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.281510 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:01:23.281786 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.281796 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.282919 (+ 1123us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.282925 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.283766 (+ 841us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.283775 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.283918 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.283969 (+ 51us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.284055 (+ 86us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.284127 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":95,"threads_started":1}
I20260429 08:01:23.284598 23737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.280046 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:36425 (local address 127.0.0.1:52696)
0429 08:01:23.281049 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.281065 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.281152 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.281733 (+ 581us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.281736 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.281751 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:01:23.281960 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.281965 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.283352 (+ 1387us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.283355 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.284238 (+ 883us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.284258 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.284353 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.284366 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.284412 (+ 46us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.284460 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":122,"threads_started":1}
I20260429 08:01:23.285024 23735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.279954 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.12.12.129:46021)
0429 08:01:23.280542 (+ 588us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.280550 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.281030 (+ 480us) server_negotiation.cc:408] Connection header received
0429 08:01:23.281126 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.281131 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.281314 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.281473 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:01:23.281914 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.282804 (+ 890us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.283918 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.284454 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.284568 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.284631 (+ 63us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.284717 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":145,"threads_started":1}
I20260429 08:01:23.285748 23739 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.280713 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.12.12.130:46793)
0429 08:01:23.281195 (+ 482us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.281201 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.281531 (+ 330us) server_negotiation.cc:408] Connection header received
0429 08:01:23.281612 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.281617 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.281748 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.281863 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:01:23.282389 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.283500 (+ 1111us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.284851 (+ 1351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.285371 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.285481 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.285543 (+ 62us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.285617 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":130,"threads_started":1}
I20260429 08:01:23.285754 23738 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.280680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.12.12.131:36425)
0429 08:01:23.281315 (+ 635us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.281321 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.281339 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:01:23.281421 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.281426 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.281596 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.281735 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:01:23.282111 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.283221 (+ 1110us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.284898 (+ 1677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.285407 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.285481 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.285543 (+ 62us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.285617 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":91,"threads_started":1}
I20260429 08:01:23.284739 23736 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.280356 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46793 (local address 127.0.0.1:53718)
0429 08:01:23.281357 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.281370 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.281445 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.282002 (+ 557us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.282005 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.282023 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:01:23.282261 (+ 238us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.282268 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.283616 (+ 1348us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.283619 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.284433 (+ 814us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.284440 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.284524 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.284536 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.284578 (+ 42us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.284623 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":82,"threads_started":1}
I20260429 08:01:23.287777 23385 tablet_service.cc:1511] Processing CreateTablet for tablet fb5758f986c849a7a971cb2dda44ef3e (DEFAULT_TABLE table=test-table [id=0a712d6f56b440c3a6acac3ae8595d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:01:23.288256 23657 tablet_service.cc:1511] Processing CreateTablet for tablet fb5758f986c849a7a971cb2dda44ef3e (DEFAULT_TABLE table=test-table [id=0a712d6f56b440c3a6acac3ae8595d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:01:23.288689 23385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb5758f986c849a7a971cb2dda44ef3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:23.289135 23657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb5758f986c849a7a971cb2dda44ef3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:23.294098 23740 tablet_bootstrap.cc:492] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: Bootstrap starting.
I20260429 08:01:23.294497 23741 tablet_bootstrap.cc:492] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Bootstrap starting.
I20260429 08:01:23.294808 23521 tablet_service.cc:1511] Processing CreateTablet for tablet fb5758f986c849a7a971cb2dda44ef3e (DEFAULT_TABLE table=test-table [id=0a712d6f56b440c3a6acac3ae8595d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:01:23.295624 23521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb5758f986c849a7a971cb2dda44ef3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:01:23.296478 23741 tablet_bootstrap.cc:654] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:23.297216 23741 log.cc:826] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:23.299101 23741 tablet_bootstrap.cc:492] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: No bootstrap required, opened a new log
I20260429 08:01:23.299296 23741 ts_tablet_manager.cc:1403] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260429 08:01:23.299431 23740 tablet_bootstrap.cc:654] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:23.300551 23740 log.cc:826] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:23.300936 23743 tablet_bootstrap.cc:492] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: Bootstrap starting.
I20260429 08:01:23.302973 23740 tablet_bootstrap.cc:492] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: No bootstrap required, opened a new log
I20260429 08:01:23.303109 23743 tablet_bootstrap.cc:654] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: Neither blocks nor log segments found. Creating new log.
I20260429 08:01:23.303195 23740 ts_tablet_manager.cc:1403] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260429 08:01:23.303655 23741 raft_consensus.cc:359] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.303860 23741 raft_consensus.cc:385] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:23.303897 23743 log.cc:826] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: Log is configured to *not* fsync() on all Append() calls
I20260429 08:01:23.303931 23741 raft_consensus.cc:740] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1372b069f81a46b9b07cd83503f5928e, State: Initialized, Role: FOLLOWER
I20260429 08:01:23.304373 23741 consensus_queue.cc:260] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.305109 23723 heartbeater.cc:499] Master 127.12.12.190:34995 was elected leader, sending a full tablet report...
I20260429 08:01:23.305337 23741 ts_tablet_manager.cc:1434] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260429 08:01:23.305833 23743 tablet_bootstrap.cc:492] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: No bootstrap required, opened a new log
I20260429 08:01:23.305995 23743 ts_tablet_manager.cc:1403] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 08:01:23.306850 23740 raft_consensus.cc:359] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.307070 23740 raft_consensus.cc:385] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:23.307171 23740 raft_consensus.cc:740] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7729d3c6f84c48a3ddfc29ebfd1dc1, State: Initialized, Role: FOLLOWER
I20260429 08:01:23.307603 23740 consensus_queue.cc:260] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.308442 23740 ts_tablet_manager.cc:1434] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260429 08:01:23.308755 23451 heartbeater.cc:499] Master 127.12.12.190:34995 was elected leader, sending a full tablet report...
I20260429 08:01:23.310091 23743 raft_consensus.cc:359] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.310344 23743 raft_consensus.cc:385] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:01:23.310400 23743 raft_consensus.cc:740] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758fc064c26e436bafbde672e69717c8, State: Initialized, Role: FOLLOWER
I20260429 08:01:23.310829 23743 consensus_queue.cc:260] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.311606 23587 heartbeater.cc:499] Master 127.12.12.190:34995 was elected leader, sending a full tablet report...
I20260429 08:01:23.311725 23743 ts_tablet_manager.cc:1434] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260429 08:01:23.318936 23452 tablet.cc:2404] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:01:23.378082 23748 raft_consensus.cc:493] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:01:23.378262 23748 raft_consensus.cc:515] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.379469 23748 leader_election.cc:290] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b7729d3c6f84c48a3ddfc29ebfd1dc1 (127.12.12.129:46021), 1372b069f81a46b9b07cd83503f5928e (127.12.12.131:36425)
I20260429 08:01:23.382812 23589 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.379716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:36425 (local address 127.12.12.130:51177)
0429 08:01:23.379861 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.379879 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.379983 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.380235 (+ 252us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.380237 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.380252 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:01:23.380512 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.380520 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.381531 (+ 1011us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.381535 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.382345 (+ 810us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.382354 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.382522 (+ 168us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.382541 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.382603 (+ 62us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.382657 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260429 08:01:23.383133 23738 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.379867 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:51177 (local address 127.12.12.131:36425)
0429 08:01:23.379998 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.380003 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.380018 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:01:23.380075 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.380077 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.380121 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.380188 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:01:23.380665 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.381363 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.382500 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.382829 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.382858 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.382932 (+ 74us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.383011 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 08:01:23.383839 23677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb5758f986c849a7a971cb2dda44ef3e" candidate_uuid: "758fc064c26e436bafbde672e69717c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1372b069f81a46b9b07cd83503f5928e" is_pre_election: true
I20260429 08:01:23.383968 23749 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.379716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:46021 (local address 127.12.12.130:59437)
0429 08:01:23.380568 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.380582 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:01:23.380673 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:01:23.380914 (+ 241us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:01:23.380916 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:01:23.380931 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:01:23.381144 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.381149 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.381983 (+ 834us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.381987 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:01:23.382871 (+ 884us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:01:23.382882 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.383657 (+ 775us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.383670 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 08:01:23.383771 (+ 101us) client_negotiation.cc:241] Negotiation successful
0429 08:01:23.383824 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":528,"thread_start_us":117,"threads_started":1}
I20260429 08:01:23.383981 23735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.379840 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:59437 (local address 127.12.12.129:46021)
0429 08:01:23.379970 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.379975 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.380667 (+ 692us) server_negotiation.cc:408] Connection header received
0429 08:01:23.380745 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.380749 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.380799 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.380897 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:01:23.381283 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.381850 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.383065 (+ 1215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.383547 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.383583 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.383801 (+ 218us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.383866 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260429 08:01:23.384153 23677 raft_consensus.cc:2468] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758fc064c26e436bafbde672e69717c8 in term 0.
I20260429 08:01:23.384636 23476 leader_election.cc:304] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1372b069f81a46b9b07cd83503f5928e, 758fc064c26e436bafbde672e69717c8; no voters:
I20260429 08:01:23.384877 23405 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb5758f986c849a7a971cb2dda44ef3e" candidate_uuid: "758fc064c26e436bafbde672e69717c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" is_pre_election: true
I20260429 08:01:23.384917 23748 raft_consensus.cc:2804] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:01:23.384994 23748 raft_consensus.cc:493] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:01:23.385059 23748 raft_consensus.cc:3060] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:23.385191 23405 raft_consensus.cc:2468] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758fc064c26e436bafbde672e69717c8 in term 0.
I20260429 08:01:23.386061 23748 raft_consensus.cc:515] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.386432 23748 leader_election.cc:290] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8b7729d3c6f84c48a3ddfc29ebfd1dc1 (127.12.12.129:46021), 1372b069f81a46b9b07cd83503f5928e (127.12.12.131:36425)
I20260429 08:01:23.386759 23405 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb5758f986c849a7a971cb2dda44ef3e" candidate_uuid: "758fc064c26e436bafbde672e69717c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1"
I20260429 08:01:23.386812 23677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb5758f986c849a7a971cb2dda44ef3e" candidate_uuid: "758fc064c26e436bafbde672e69717c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1372b069f81a46b9b07cd83503f5928e"
I20260429 08:01:23.386895 23405 raft_consensus.cc:3060] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:23.386955 23677 raft_consensus.cc:3060] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:01:23.387853 23677 raft_consensus.cc:2468] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758fc064c26e436bafbde672e69717c8 in term 1.
I20260429 08:01:23.387960 23405 raft_consensus.cc:2468] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758fc064c26e436bafbde672e69717c8 in term 1.
I20260429 08:01:23.388140 23476 leader_election.cc:304] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1372b069f81a46b9b07cd83503f5928e, 758fc064c26e436bafbde672e69717c8; no voters:
I20260429 08:01:23.388410 23748 raft_consensus.cc:2804] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:01:23.388589 23748 raft_consensus.cc:697] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 1 LEADER]: Becoming Leader. State: Replica: 758fc064c26e436bafbde672e69717c8, State: Running, Role: LEADER
I20260429 08:01:23.388934 23748 consensus_queue.cc:237] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } }
I20260429 08:01:23.392064 23254 catalog_manager.cc:5671] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758fc064c26e436bafbde672e69717c8 (127.12.12.130). New cstate: current_term: 1 leader_uuid: "758fc064c26e436bafbde672e69717c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758fc064c26e436bafbde672e69717c8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 46793 } health_report { overall_health: HEALTHY } } }
I20260429 08:01:23.408059 23739 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.405145 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.12.12.130:46793)
0429 08:01:23.405296 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 08:01:23.405303 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:01:23.405320 (+ 17us) server_negotiation.cc:408] Connection header received
0429 08:01:23.405400 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:01:23.405403 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:01:23.405451 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:01:23.405560 (+ 109us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:01:23.405929 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.406393 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:01:23.407051 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:01:23.407248 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:01:23.407324 (+ 76us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:01:23.407677 (+ 353us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:01:23.407800 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 08:01:23.407878 (+ 78us) server_negotiation.cc:300] Negotiation successful
0429 08:01:23.407921 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 08:01:23.413249 23677 raft_consensus.cc:1275] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 1 FOLLOWER]: Refusing update from remote peer 758fc064c26e436bafbde672e69717c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:01:23.413286 23405 raft_consensus.cc:1275] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 1 FOLLOWER]: Refusing update from remote peer 758fc064c26e436bafbde672e69717c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:01:23.413933 23748 consensus_queue.cc:1048] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7729d3c6f84c48a3ddfc29ebfd1dc1" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:01:23.414103 23750 consensus_queue.cc:1048] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1372b069f81a46b9b07cd83503f5928e" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 36425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:01:23.428669 23251 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:01:23.428866 23251 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:01:23.431176 23251 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 159f05743a814237a443a61fdd0768e1: Sending DeleteTablet for 3 replicas of tablet fb5758f986c849a7a971cb2dda44ef3e
I20260429 08:01:23.431859 23385 tablet_service.cc:1558] Processing DeleteTablet for tablet fb5758f986c849a7a971cb2dda44ef3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:01:23 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:33068
I20260429 08:01:23.431885 23657 tablet_service.cc:1558] Processing DeleteTablet for tablet fb5758f986c849a7a971cb2dda44ef3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:01:23 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:52696
I20260429 08:01:23.432313 23763 tablet_replica.cc:333] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1: stopping tablet replica
I20260429 08:01:23.432536 23763 raft_consensus.cc:2243] T fb5758f986c849a7a971cb2dda44ef3e P 8b7729d3c6f84c48a3ddfc29ebfd1dc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:01:23.432720 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 23322
I20260429 08:01:23.432893 23521 tablet_service.cc:1558] Processing DeleteTablet for tablet fb5758f986c849a7a971cb2dda44ef3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:01:23 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:53718
I20260429 08:01:23.433713 23764 tablet_replica.cc:333] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: stopping tablet replica
I20260429 08:01:23.433884 23764 raft_consensus.cc:2243] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:01:23.434073 23764 raft_consensus.cc:2272] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:01:23.435141 23764 ts_tablet_manager.cc:1916] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:01:23.436987 23764 ts_tablet_manager.cc:1929] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:01:23.437116 23764 log.cc:1199] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/fb5758f986c849a7a971cb2dda44ef3e
I20260429 08:01:23.437444 23764 ts_tablet_manager.cc:1950] T fb5758f986c849a7a971cb2dda44ef3e P 1372b069f81a46b9b07cd83503f5928e: Deleting consensus metadata
I20260429 08:01:23.438428 23244 catalog_manager.cc:5002] TS 1372b069f81a46b9b07cd83503f5928e (127.12.12.131:36425): tablet fb5758f986c849a7a971cb2dda44ef3e (table test-table [id=0a712d6f56b440c3a6acac3ae8595d57]) successfully deleted
I20260429 08:01:23.440995 23765 tablet_replica.cc:333] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8: stopping tablet replica
W20260429 08:01:23.441159 23243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.129:46021 (error 108)
I20260429 08:01:23.441265 23765 raft_consensus.cc:2243] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:01:23.441397 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 23458
I20260429 08:01:23.441694 23765 raft_consensus.cc:2272] T fb5758f986c849a7a971cb2dda44ef3e P 758fc064c26e436bafbde672e69717c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:01:23.443477 23737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.443216 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:46021 (local address 127.0.0.1:33080)
0429 08:01:23.443339 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.443391 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:46021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260429 08:01:23.443620 23243 catalog_manager.cc:4729] TS 8b7729d3c6f84c48a3ddfc29ebfd1dc1 (127.12.12.129:46021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb5758f986c849a7a971cb2dda44ef3e: Network error: Client connection negotiation failed: client connection to 127.12.12.129:46021: connect: Connection refused (error 111)
I20260429 08:01:23.447190 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 23594
I20260429 08:01:23.448895 23737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:01:23.447456 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:46793 (local address 127.0.0.1:53734)
0429 08:01:23.448712 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0429 08:01:23.448780 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.130:46793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1177}
I20260429 08:01:23.452767 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 23226
2026-04-29T08:01:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3232 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:01:23 dist-test-slave-nhqz krb5kdc[23769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:01:23 dist-test-slave-nhqz krb5kdc[23769](info): set up 2 sockets
Apr 29 08:01:23 dist-test-slave-nhqz krb5kdc[23769](info): commencing operation
krb5kdc: starting...
W20260429 08:01:25.454341 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:01:25 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:01:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:01:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/rangeradmin_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/rangeradmin_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/rangerlookup_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/rangerlookup_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
I20260429 08:01:25.612948 12338 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPttFKW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/postgres -l logfile start
2026-04-29 08:01:28.190 UTC [23800] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-29 08:01:28.190 UTC [23800] LOG: listening on IPv4 address "127.12.12.148", port 45305
2026-04-29 08:01:28.194 UTC [23800] LOG: listening on Unix socket "/tmp/.s.PGSQL.45305"
2026-04-29 08:01:28.199 UTC [23805] LOG: database system was shut down at 2026-04-29 08:01:26 UTC
2026-04-29 08:01:28.204 UTC [23800] LOG: database system is ready to accept connections
I20260429 08:01:30.186051 12338 mini_postgres.cc:96] Postgres bound to 45305
2026-04-29 08:01:30.192 UTC [23811] FATAL: database "slave" does not exist
127.12.12.148:45305 - accepting connections
I20260429 08:01:30.192710 12338 mini_ranger.cc:161] Starting Ranger...
I20260429 08:01:30.210285 12338 mini_ranger.cc:84] Created miniranger Postgres user
I20260429 08:01:30.281034 12338 mini_ranger.cc:87] Created ranger Postgres database
I20260429 08:01:30.281194 12338 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-admin
2026-04-29 08:01:30,572 [I] DB FLAVOR :POSTGRES
2026-04-29 08:01:30,572 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 08:01:30,572 [I] Checking connection..
2026-04-29 08:01:30,572 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-29 08:01:30,925 [I] Checking connection passed.
2026-04-29 08:01:30,925 [I] --------- Verifying version history table ---------
2026-04-29 08:01:30,925 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 08:01:31,261 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 08:01:31,262 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 08:01:31,571 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 08:01:31,571 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-29 08:01:31,571 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-29 08:01:31.850 UTC [23924] WARNING: there is no transaction in progress
2026-04-29 08:01:31,873 [I] create_dbversion_catalog.sql file imported successfully
2026-04-29 08:01:31,874 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 08:01:32,187 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-29 08:01:32,188 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 08:01:32,188 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-29 08:01:32,508 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-29 08:01:32,850 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nhqz.c.gcp-upstream.internal','N') ;"
2026-04-29 08:01:33,198 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 08:01:33,505 [I] Table x_portal_user does not exist in database ranger
2026-04-29 08:01:33,505 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 08:01:33,845 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-29 08:01:33,846 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 08:01:34,120 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 08:01:34,467 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-29 08:01:34,468 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-29 08:01:35.371 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.382 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.394 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.404 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.415 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.425 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.470 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.475 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.488 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.497 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.506 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.515 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.523 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.527 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.531 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35.535 UTC [24122] WARNING: there is no transaction in progress
2026-04-29 08:01:35,936 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-29 08:01:35,936 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-29 08:01:36,217 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-29 08:01:36,217 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 08:01:36,555 [I] Table x_portal_user already exists in database 'ranger'
2026-04-29 08:01:36,556 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 08:01:36,884 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-29 08:01:36,884 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 08:01:37,244 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 08:01:37,603 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nhqz.c.gcp-upstream.internal';"
2026-04-29 08:01:37,950 [I] CORE_DB_SCHEMA import status has been updated
2026-04-29 08:01:37,950 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-29 08:01:38,279 [I] DB_PATCHES have already been applied
I20260429 08:01:38.285744 12338 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskPttFKW/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260429 08:01:38.285801 12338 mini_ranger.cc:193] Using host: 127.12.12.148
I20260429 08:01:38.288759 12338 mini_ranger.cc:239] Ranger admin URL: http://127.12.12.148:39657
Apr 29, 2026 8:01:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 8:01:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 8:01:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp
Apr 29, 2026 8:01:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 29, 2026 8:01:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 29, 2026 8:01:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp].
Apr 29, 2026 8:01:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260429 08:01:40.576376 12338 mini_ranger.cc:160] Time spent starting Ranger: real 10.384s user 0.003s sys 0.007s
Apr 29, 2026 8:02:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 29, 2026 8:02:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Apr 29, 2026 8:02:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 29, 2026 8:02:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 29, 2026 8:02:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260429 08:02:04.838444 12338 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/rangerkms_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/rangerkms_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.12.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Apr 29 08:02:04 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449724, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260429 08:02:04.939311 12338 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260429 08:02:04.951706 12338 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260429 08:02:05.079413 12338 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260429 08:02:05.079511 12338 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms
I20260429 08:02:05.079550 12338 mini_ranger_kms.cc:227] Using postgres at 127.12.12.148:45305
2026-04-29 08:02:05,302 [I] DB FLAVOR :POSTGRES
2026-04-29 08:02:05,302 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 08:02:05,302 [I] Checking connection
2026-04-29 08:02:05,302 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-04-29 08:02:05,694 [I] connection success
2026-04-29 08:02:05,694 [I] --------- Verifying Ranger DB tables ---------
2026-04-29 08:02:05,694 [I] Verifying table ranger_masterkey in database rangerkms
2026-04-29 08:02:05,694 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-04-29 08:02:06,041 [I] Table ranger_masterkey does not exist in database rangerkms
2026-04-29 08:02:06,041 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 08:02:06,041 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-04-29 08:02:06,041 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:45305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-04-29 08:02:06,377 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260429 08:02:06.889115 12338 mini_ranger_kms.cc:326] Created kms service
I20260429 08:02:07.119874 12338 mini_ranger_kms.cc:342] Created kudu user
I20260429 08:02:07.167243 12338 mini_ranger_kms.cc:359] Created rangerkms user
Apr 29 08:02:07 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449724, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:02:07.638451 12338 mini_ranger_kms.cc:400] Added ranger policy
I20260429 08:02:07.638604 12338 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260429 08:02:07.638625 12338 mini_ranger_kms.cc:240] Using host: 127.12.12.148
I20260429 08:02:07.642655 12338 mini_ranger_kms.cc:292] Ranger KMS PID: 24422
I20260429 08:02:07.642771 12338 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.12.12.148:60535
Apr 29, 2026 8:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 8:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 8:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Apr 29, 2026 8:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/ews/webapp] .....
Apr 29, 2026 8:02:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-kms/ews/webapp].
Apr 29, 2026 8:02:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260429 08:02:09.913482 12338 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.974s user 0.001s sys 0.192s
I20260429 08:02:09.913614 12338 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260429 08:02:09.913692 12338 mini_ranger_kms.cc:417] 127.12.12.148:60535/kms/v1/keys
Apr 29 08:02:10 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449730, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Apr 29 08:02:10 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449730, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.148@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
Apr 29 08:02:11 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449730, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.148@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
Apr 29 08:02:13 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:02:13.377311 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:41057
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:41057
--ranger_config_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:60535/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:02:13.480046 24490 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:02:13.480341 24490 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:02:13.480427 24490 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:02:13.483911 24490 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:02:13.484018 24490 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:02:13.484063 24490 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:02:13.484127 24490 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:02:13.484193 24490 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:02:13.489058 24490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.12.148:60535/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:41057
--ranger_config_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:41057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:02:13.490240 24490 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:02:13.491114 24490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:02:13.496939 24495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:13.496950 24498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:13.496937 24496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:02:13.497363 24490 server_base.cc:1061] running on GCE node
I20260429 08:02:13.497965 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:02:13.499025 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:02:13.500211 24490 hybrid_clock.cc:648] HybridClock initialized: now 1777449733500196 us; error 56 us; skew 500 ppm
Apr 29 08:02:13 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449733, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:02:13.503561 24490 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:02:13.504695 24490 webserver.cc:492] Webserver started at http://127.12.12.190:38561/ using document root <none> and password file <none>
I20260429 08:02:13.505295 24490 fs_manager.cc:362] Metadata directory not provided
I20260429 08:02:13.505370 24490 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:02:13.505580 24490 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:02:13 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449733, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:02:13.550629 24490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "1e01185d165e436db285add8f325d7d9"
format_stamp: "Formatted at 2026-04-29 08:02:13 on dist-test-slave-nhqz"
server_key: "3b632989e900f9024eea3d4f9b09c871"
server_key_iv: "3bfbb5e195452a3f03b776c39d11f152"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:13.551190 24490 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "1e01185d165e436db285add8f325d7d9"
format_stamp: "Formatted at 2026-04-29 08:02:13 on dist-test-slave-nhqz"
server_key: "3b632989e900f9024eea3d4f9b09c871"
server_key_iv: "3bfbb5e195452a3f03b776c39d11f152"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:13.555375 24490 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260429 08:02:13.574522 24506 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:13.575945 24490 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260429 08:02:13.576090 24490 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "1e01185d165e436db285add8f325d7d9"
format_stamp: "Formatted at 2026-04-29 08:02:13 on dist-test-slave-nhqz"
server_key: "3b632989e900f9024eea3d4f9b09c871"
server_key_iv: "3bfbb5e195452a3f03b776c39d11f152"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:13.576231 24490 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:02:13.606634 24490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:02:13.609112 24490 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:02:13.609272 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:02:13.616609 24490 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:41057
I20260429 08:02:13.616609 24558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:41057 every 8 connection(s)
I20260429 08:02:13.617660 24490 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:02:13.620237 24559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:02:13.624789 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 24490
I20260429 08:02:13.624966 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:02:13.625195 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9: Bootstrap starting.
I20260429 08:02:13.625244 12338 external_mini_cluster.cc:1468] Setting key 114903a3c32ad32864c01765b123e25b
I20260429 08:02:13.627650 24559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9: Neither blocks nor log segments found. Creating new log.
I20260429 08:02:13.628438 24559 log.cc:826] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9: Log is configured to *not* fsync() on all Append() calls
I20260429 08:02:13.630458 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9: No bootstrap required, opened a new log
Apr 29 08:02:13 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:02:13.633767 24559 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01185d165e436db285add8f325d7d9" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41057 } }
I20260429 08:02:13.633985 24559 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:02:13.634063 24559 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e01185d165e436db285add8f325d7d9, State: Initialized, Role: FOLLOWER
I20260429 08:02:13.634475 24559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01185d165e436db285add8f325d7d9" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41057 } }
I20260429 08:02:13.634611 24559 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:02:13.634687 24559 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:02:13.634793 24559 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:02:13.635556 24559 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01185d165e436db285add8f325d7d9" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41057 } }
I20260429 08:02:13.635864 24559 leader_election.cc:304] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e01185d165e436db285add8f325d7d9; no voters:
I20260429 08:02:13.636142 24559 leader_election.cc:290] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:02:13.636212 24564 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:02:13.636395 24564 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [term 1 LEADER]: Becoming Leader. State: Replica: 1e01185d165e436db285add8f325d7d9, State: Running, Role: LEADER
I20260429 08:02:13.636713 24564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01185d165e436db285add8f325d7d9" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41057 } }
I20260429 08:02:13.637247 24559 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:02:13.638154 24562 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:13.626456 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.12.12.190:41057)
0429 08:02:13.626905 (+ 449us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:13.626917 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:13.626953 (+ 36us) server_negotiation.cc:408] Connection header received
0429 08:02:13.627605 (+ 652us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:13.627630 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:13.627971 (+ 341us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:13.628302 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:02:13.629307 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:13.630402 (+ 1095us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:13.631074 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:13.631351 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:13.633644 (+ 2293us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:02:13.633664 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:02:13.633677 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:02:13.633706 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:02:13.635477 (+ 1771us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:13.635948 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:13.635953 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:13.635957 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:13.636049 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:13.636315 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:13.636318 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:13.636320 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:13.636742 (+ 422us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:02:13.636963 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:13.637203 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 08:02:13.637394 (+ 191us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":137,"threads_started":1}
I20260429 08:02:13.638293 24565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e01185d165e436db285add8f325d7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01185d165e436db285add8f325d7d9" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41057 } } }
I20260429 08:02:13.638401 24565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [sys.catalog]: This master's current role is: LEADER
I20260429 08:02:13.638736 24572 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:02:13.639653 24566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e01185d165e436db285add8f325d7d9. Latest consensus state: current_term: 1 leader_uuid: "1e01185d165e436db285add8f325d7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e01185d165e436db285add8f325d7d9" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 41057 } } }
I20260429 08:02:13.639732 24566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9 [sys.catalog]: This master's current role is: LEADER
I20260429 08:02:13.641594 24572 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:02:13.641947 24559 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260429 08:02:13.650699 24572 catalog_manager.cc:1357] Generated new cluster ID: 515340ae378d4ed393529798f107abc6
I20260429 08:02:13.650802 24572 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:02:13.667048 24572 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:02:13.667906 24572 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:02:13.676168 24572 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9: Generated new TSK 0
I20260429 08:02:13.676899 24572 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:02:14.396103 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:60535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:41057
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:02:14.513064 24618 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:02:14.513350 24618 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:02:14.513485 24618 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:02:14.516924 24618 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:02:14.517009 24618 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:02:14.517140 24618 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:02:14.521697 24618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.12.148:60535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:41057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:02:14.522924 24618 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:02:14.523783 24618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:02:14.531737 24626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:14.531745 24623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:14.532174 24624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:02:14.534782 24618 server_base.cc:1061] running on GCE node
I20260429 08:02:14.535203 24618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:02:14.535869 24618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:02:14.539139 24618 hybrid_clock.cc:648] HybridClock initialized: now 1777449734539104 us; error 49 us; skew 500 ppm
Apr 29 08:02:14 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449734, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:02:14.542907 24618 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:02:14.544346 24618 webserver.cc:492] Webserver started at http://127.12.12.129:43875/ using document root <none> and password file <none>
I20260429 08:02:14.545147 24618 fs_manager.cc:362] Metadata directory not provided
I20260429 08:02:14.545276 24618 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:02:14.545557 24618 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:02:14 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449734, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:02:14.573084 24618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "bda8038c9a174145b0a2e6272d21719f"
format_stamp: "Formatted at 2026-04-29 08:02:14 on dist-test-slave-nhqz"
server_key: "40ddbe552617a09c3758978b3802978e"
server_key_iv: "944dc505777ab3931f268e0cb8b652bb"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:14.573741 24618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "bda8038c9a174145b0a2e6272d21719f"
format_stamp: "Formatted at 2026-04-29 08:02:14 on dist-test-slave-nhqz"
server_key: "40ddbe552617a09c3758978b3802978e"
server_key_iv: "944dc505777ab3931f268e0cb8b652bb"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:14.590394 24618 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260429 08:02:14.638325 24635 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:14.641168 24618 fs_manager.cc:730] Time spent opening block manager: real 0.021s user 0.003s sys 0.000s
I20260429 08:02:14.641324 24618 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "bda8038c9a174145b0a2e6272d21719f"
format_stamp: "Formatted at 2026-04-29 08:02:14 on dist-test-slave-nhqz"
server_key: "40ddbe552617a09c3758978b3802978e"
server_key_iv: "944dc505777ab3931f268e0cb8b652bb"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:14.641444 24618 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:02:14.683694 24618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:02:14.707923 24618 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:02:14.708230 24618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:02:14.708940 24618 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:02:14.710196 24618 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:02:14.710307 24618 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:14.710433 24618 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:02:14.710520 24618 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:14.740435 24618 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:43985
I20260429 08:02:14.741933 24618 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:02:14.752233 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 24618
I20260429 08:02:14.752383 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:02:14.752715 12338 external_mini_cluster.cc:1468] Setting key 6af7947f0c3d8ab61d72bda11228bda4
I20260429 08:02:14.752699 24751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:43985 every 8 connection(s)
Apr 29 08:02:14 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449734, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
I20260429 08:02:14.786026 24755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:14.755975 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:60673 (local address 127.12.12.190:41057)
0429 08:02:14.766950 (+ 10975us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:14.766954 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:14.766969 (+ 15us) server_negotiation.cc:408] Connection header received
0429 08:02:14.767019 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:14.767023 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:14.767083 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:14.767180 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:02:14.768793 (+ 1613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:14.769640 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:14.773737 (+ 4097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:14.773954 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:14.781425 (+ 7471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:02:14.781456 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:02:14.781460 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:02:14.781499 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:02:14.783562 (+ 2063us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:14.784273 (+ 711us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:14.784276 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:14.784278 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:14.784332 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:14.784681 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:14.784684 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:14.784686 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:14.784886 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:02:14.784974 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:14.785699 (+ 725us) server_negotiation.cc:300] Negotiation successful
0429 08:02:14.785833 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10868,"thread_start_us":85,"threads_started":1}
I20260429 08:02:14.787765 24754 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:14.754923 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41057 (local address 127.12.12.129:60673)
0429 08:02:14.765223 (+ 10300us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:14.765294 (+ 71us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:14.766425 (+ 1131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:14.767539 (+ 1114us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:14.767552 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:14.767924 (+ 372us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:02:14.768615 (+ 691us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:14.768631 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:14.772267 (+ 3636us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:14.772272 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:14.772932 (+ 660us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:14.772942 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:14.773192 (+ 250us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:14.774315 (+ 1123us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:02:14.774341 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:02:14.781131 (+ 6790us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:02:14.783745 (+ 2614us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:02:14.783753 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:02:14.783770 (+ 17us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:02:14.784127 (+ 357us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:02:14.784443 (+ 316us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:02:14.784446 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:02:14.784448 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:02:14.784580 (+ 132us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:02:14.785056 (+ 476us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:02:14.785062 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:02:14.785385 (+ 323us) client_negotiation.cc:770] Sending connection context
0429 08:02:14.786566 (+ 1181us) client_negotiation.cc:241] Negotiation successful
0429 08:02:14.786815 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10007,"thread_start_us":119,"threads_started":1}
I20260429 08:02:14.789307 24752 heartbeater.cc:344] Connected to a master server at 127.12.12.190:41057
I20260429 08:02:14.789570 24752 heartbeater.cc:461] Registering TS with master...
I20260429 08:02:14.790105 24752 heartbeater.cc:507] Master 127.12.12.190:41057 requested a full tablet report, sending...
I20260429 08:02:14.791775 24523 ts_manager.cc:194] Registered new tserver with Master: bda8038c9a174145b0a2e6272d21719f (127.12.12.129:43985)
I20260429 08:02:14.793340 24523 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:60673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Apr 29 08:02:14 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449734, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:02:14.855885 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:60535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:41057
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:02:14.989269 24760 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:02:14.989518 24760 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:02:14.989615 24760 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:02:14.993088 24760 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:02:14.993175 24760 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:02:14.993294 24760 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:02:14.998188 24760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.12.148:60535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:41057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:02:14.999650 24760 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:02:15.000639 24760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:02:15.010126 24767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:15.010306 24766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:15.010672 24769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:02:15.011780 24760 server_base.cc:1061] running on GCE node
I20260429 08:02:15.012363 24760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:02:15.013113 24760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:02:15.014329 24760 hybrid_clock.cc:648] HybridClock initialized: now 1777449735014296 us; error 48 us; skew 500 ppm
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449735, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:02:15.018221 24760 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:02:15.021836 24760 webserver.cc:492] Webserver started at http://127.12.12.130:36951/ using document root <none> and password file <none>
I20260429 08:02:15.022684 24760 fs_manager.cc:362] Metadata directory not provided
I20260429 08:02:15.022814 24760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:02:15.023139 24760 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449735, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:02:15.038638 24760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "0f1bff319c33451e9611edc45f06bdd8"
format_stamp: "Formatted at 2026-04-29 08:02:15 on dist-test-slave-nhqz"
server_key: "da9d3e6e93f6f4ef912b08f0da5916d2"
server_key_iv: "ab073cd4dacd794cadc94965e38d0deb"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:15.039214 24760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "0f1bff319c33451e9611edc45f06bdd8"
format_stamp: "Formatted at 2026-04-29 08:02:15 on dist-test-slave-nhqz"
server_key: "da9d3e6e93f6f4ef912b08f0da5916d2"
server_key_iv: "ab073cd4dacd794cadc94965e38d0deb"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:15.056480 24760 fs_manager.cc:696] Time spent creating directory manager: real 0.017s user 0.000s sys 0.003s
I20260429 08:02:15.102794 24776 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:15.104417 24760 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260429 08:02:15.104552 24760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "0f1bff319c33451e9611edc45f06bdd8"
format_stamp: "Formatted at 2026-04-29 08:02:15 on dist-test-slave-nhqz"
server_key: "da9d3e6e93f6f4ef912b08f0da5916d2"
server_key_iv: "ab073cd4dacd794cadc94965e38d0deb"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:15.104672 24760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:02:15.121976 24760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:02:15.125483 24760 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:02:15.125717 24760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:02:15.126401 24760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:02:15.127467 24760 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:02:15.127573 24760 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:15.127667 24760 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:02:15.127734 24760 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:15.141911 24760 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:37159
I20260429 08:02:15.143327 24760 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:02:15.145032 24889 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:37159 every 8 connection(s)
I20260429 08:02:15.145215 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 24760
I20260429 08:02:15.145401 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:02:15.145706 12338 external_mini_cluster.cc:1468] Setting key f0b71444b9dcdec5bb0122daf0733cf8
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449735, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:02:15.163692 24755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.150358 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.130:41179 (local address 127.12.12.190:41057)
0429 08:02:15.150532 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:15.150537 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:15.150857 (+ 320us) server_negotiation.cc:408] Connection header received
0429 08:02:15.152092 (+ 1235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:15.152096 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:15.152153 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:15.152207 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:02:15.154785 (+ 2578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.155565 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.156444 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.156632 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.159427 (+ 2795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:02:15.159452 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:02:15.159455 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:02:15.159486 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:02:15.161393 (+ 1907us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:15.162004 (+ 611us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:15.162008 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:15.162010 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:15.162061 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:15.162410 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:15.162414 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:15.162416 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:15.162597 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:02:15.162676 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:15.163374 (+ 698us) server_negotiation.cc:300] Negotiation successful
0429 08:02:15.163514 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260429 08:02:15.165402 24893 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.149424 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41057 (local address 127.12.12.130:41179)
0429 08:02:15.150730 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:15.150771 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:15.151749 (+ 978us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:15.152608 (+ 859us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:15.152619 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:15.153033 (+ 414us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:02:15.153702 (+ 669us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.153722 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.155717 (+ 1995us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.155721 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:15.156296 (+ 575us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.156305 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.156889 (+ 584us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.157531 (+ 642us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:02:15.157555 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:02:15.159231 (+ 1676us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:02:15.161536 (+ 2305us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:02:15.161543 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:02:15.161557 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:02:15.161885 (+ 328us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:02:15.162163 (+ 278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:02:15.162166 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:02:15.162168 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:02:15.162289 (+ 121us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:02:15.162743 (+ 454us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:02:15.162749 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:02:15.163071 (+ 322us) client_negotiation.cc:770] Sending connection context
0429 08:02:15.164229 (+ 1158us) client_negotiation.cc:241] Negotiation successful
0429 08:02:15.164481 (+ 252us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":129,"threads_started":1}
I20260429 08:02:15.166819 24890 heartbeater.cc:344] Connected to a master server at 127.12.12.190:41057
I20260429 08:02:15.167061 24890 heartbeater.cc:461] Registering TS with master...
I20260429 08:02:15.167673 24890 heartbeater.cc:507] Master 127.12.12.190:41057 requested a full tablet report, sending...
I20260429 08:02:15.168846 24523 ts_manager.cc:194] Registered new tserver with Master: 0f1bff319c33451e9611edc45f06bdd8 (127.12.12.130:37159)
WARNING: no policy specified for kudu/127.12.12.131@KRBTEST.COM; defaulting to no policy
I20260429 08:02:15.169553 24523 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.130@KRBTEST.COM'} at 127.12.12.130:41179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:02:15.247525 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:60535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.131:0
--local_ip_for_outbound_sockets=127.12.12.131
--webserver_interface=127.12.12.131
--webserver_port=0
--tserver_master_addrs=127.12.12.190:41057
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:42401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449734, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
W20260429 08:02:15.405253 24897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:02:15.405565 24897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:02:15.405675 24897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:02:15.410456 24897 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:02:15.410593 24897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:02:15.410738 24897 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.131
I20260429 08:02:15.416666 24897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:42401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.12.148:60535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.12.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:41057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.131
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:02:15.418327 24897 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:02:15.419433 24897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:02:15.429315 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:15.429627 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:02:15.429674 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:02:15.431038 24897 server_base.cc:1061] running on GCE node
I20260429 08:02:15.431567 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:02:15.432209 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:02:15.435464 24897 hybrid_clock.cc:648] HybridClock initialized: now 1777449735435441 us; error 31 us; skew 500 ppm
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449735, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:02:15.439178 24897 init.cc:377] Logged in from keytab as kudu/127.12.12.131@KRBTEST.COM (short username kudu)
I20260429 08:02:15.440379 24897 webserver.cc:492] Webserver started at http://127.12.12.131:41523/ using document root <none> and password file <none>
I20260429 08:02:15.441033 24897 fs_manager.cc:362] Metadata directory not provided
I20260429 08:02:15.441176 24897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:02:15.441439 24897 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449735, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:02:15.455749 24897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data/instance:
uuid: "023fdfed641f48688881844d1ceb1534"
format_stamp: "Formatted at 2026-04-29 08:02:15 on dist-test-slave-nhqz"
server_key: "7078a5132d8ad21ba83e9e5bf8f794ca"
server_key_iv: "ebcf93ad26e62800ef6cf6a186bc43b9"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:15.456439 24897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance:
uuid: "023fdfed641f48688881844d1ceb1534"
format_stamp: "Formatted at 2026-04-29 08:02:15 on dist-test-slave-nhqz"
server_key: "7078a5132d8ad21ba83e9e5bf8f794ca"
server_key_iv: "ebcf93ad26e62800ef6cf6a186bc43b9"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:15.462277 24897 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260429 08:02:15.484938 24914 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:15.486400 24897 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260429 08:02:15.486527 24897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal
uuid: "023fdfed641f48688881844d1ceb1534"
format_stamp: "Formatted at 2026-04-29 08:02:15 on dist-test-slave-nhqz"
server_key: "7078a5132d8ad21ba83e9e5bf8f794ca"
server_key_iv: "ebcf93ad26e62800ef6cf6a186bc43b9"
server_key_version: "kuduclusterkey@0"
I20260429 08:02:15.486640 24897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:02:15.512570 24897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:02:15.532624 24897 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:02:15.537606 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:02:15.538558 24897 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:02:15.546075 24897 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:02:15.546155 24897 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:15.546224 24897 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:02:15.546250 24897 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:02:15.557130 24897 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.131:34587
I20260429 08:02:15.558434 24897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/data/info.pb
I20260429 08:02:15.563805 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 24897
I20260429 08:02:15.563946 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal/instance
I20260429 08:02:15.564222 12338 external_mini_cluster.cc:1468] Setting key 5a528f3907a0f8318214b471d2ddbee0
I20260429 08:02:15.571444 25027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.131:34587 every 8 connection(s)
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449735, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.131@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:02:15.579146 24755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.562405 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:38779 (local address 127.12.12.190:41057)
0429 08:02:15.562567 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:15.562571 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:15.563012 (+ 441us) server_negotiation.cc:408] Connection header received
0429 08:02:15.567004 (+ 3992us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:15.567008 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:15.567079 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:15.567140 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:02:15.568925 (+ 1785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.569733 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.570609 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.570795 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.573892 (+ 3097us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:02:15.573912 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:02:15.573915 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:02:15.573944 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:02:15.575638 (+ 1694us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:15.576246 (+ 608us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:15.576249 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:15.576252 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:15.576302 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:15.576651 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:15.576655 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:15.576656 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:15.576861 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:02:15.576935 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:15.578823 (+ 1888us) server_negotiation.cc:300] Negotiation successful
0429 08:02:15.578973 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 08:02:15.580801 25030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.561345 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:41057 (local address 127.12.12.131:38779)
0429 08:02:15.562831 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:15.562877 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:15.566710 (+ 3833us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:15.567609 (+ 899us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:15.567621 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:15.567980 (+ 359us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:02:15.568719 (+ 739us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.568736 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.569888 (+ 1152us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.569892 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:15.570459 (+ 567us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.570469 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.571028 (+ 559us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.571721 (+ 693us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:02:15.571745 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:02:15.573695 (+ 1950us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:02:15.575800 (+ 2105us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:02:15.575806 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:02:15.575819 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:02:15.576128 (+ 309us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:02:15.576417 (+ 289us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:02:15.576419 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:02:15.576421 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:02:15.576543 (+ 122us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:02:15.576994 (+ 451us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:02:15.577000 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:02:15.577287 (+ 287us) client_negotiation.cc:770] Sending connection context
0429 08:02:15.579604 (+ 2317us) client_negotiation.cc:241] Negotiation successful
0429 08:02:15.579850 (+ 246us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":121,"threads_started":1}
I20260429 08:02:15.582243 25028 heartbeater.cc:344] Connected to a master server at 127.12.12.190:41057
I20260429 08:02:15.582526 25028 heartbeater.cc:461] Registering TS with master...
I20260429 08:02:15.583200 25028 heartbeater.cc:507] Master 127.12.12.190:41057 requested a full tablet report, sending...
I20260429 08:02:15.584420 24523 ts_manager.cc:194] Registered new tserver with Master: 023fdfed641f48688881844d1ceb1534 (127.12.12.131:34587)
I20260429 08:02:15.585147 24523 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.131@KRBTEST.COM'} at 127.12.12.131:38779
I20260429 08:02:15.593711 12338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 08:02:15.630096 24755 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.605369 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.12.12.190:41057)
0429 08:02:15.605543 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:15.605548 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:15.609702 (+ 4154us) server_negotiation.cc:408] Connection header received
0429 08:02:15.609876 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:15.609880 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:15.609970 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:15.610085 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:02:15.613900 (+ 3815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.614708 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.624713 (+ 10005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.624968 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.626433 (+ 1465us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:02:15.626458 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:02:15.626461 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:02:15.626495 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:02:15.627982 (+ 1487us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:15.628462 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:15.628466 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:15.628468 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:15.628518 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:02:15.628773 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:02:15.628776 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:02:15.628778 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:02:15.628980 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:02:15.629110 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:15.629779 (+ 669us) server_negotiation.cc:300] Negotiation successful
0429 08:02:15.629910 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260429 08:02:15.638615 24523 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 08:02:15.642155 24523 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Apr 29 08:02:15 dist-test-slave-nhqz krb5kdc[23769](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449734, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:02:15.665939 25045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.654877 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.131:34587 (local address 127.0.0.1:53396)
0429 08:02:15.660275 (+ 5398us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:15.660288 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:15.660410 (+ 122us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:15.661583 (+ 1173us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:15.661589 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:15.661609 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:02:15.662505 (+ 896us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.662511 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.664124 (+ 1613us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.664127 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:15.664980 (+ 853us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.664989 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.665544 (+ 555us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.665598 (+ 54us) client_negotiation.cc:770] Sending connection context
0429 08:02:15.665696 (+ 98us) client_negotiation.cc:241] Negotiation successful
0429 08:02:15.665773 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5330,"thread_start_us":98,"threads_started":1}
I20260429 08:02:15.666527 25043 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.656387 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.12.12.129:43985)
0429 08:02:15.659516 (+ 3129us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:15.659523 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:15.659544 (+ 21us) server_negotiation.cc:408] Connection header received
0429 08:02:15.661137 (+ 1593us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:15.661144 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:15.661311 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:15.661434 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:02:15.662425 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.663516 (+ 1091us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.665161 (+ 1645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.665707 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.665825 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:15.666274 (+ 449us) server_negotiation.cc:300] Negotiation successful
0429 08:02:15.666388 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3021,"thread_start_us":109,"threads_started":1}
I20260429 08:02:15.667246 25044 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.656713 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.12.12.131:34587)
0429 08:02:15.660137 (+ 3424us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:15.660143 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:15.660501 (+ 358us) server_negotiation.cc:408] Connection header received
0429 08:02:15.660594 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:15.660600 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:15.660798 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:15.660959 (+ 161us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:02:15.662852 (+ 1893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.663974 (+ 1122us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.666306 (+ 2332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.666837 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.666935 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:15.666992 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 08:02:15.667063 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3303,"thread_start_us":150,"threads_started":1}
I20260429 08:02:15.668435 25041 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.657703 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:37159 (local address 127.0.0.1:36912)
0429 08:02:15.662681 (+ 4978us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:15.662693 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:15.662755 (+ 62us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:15.665617 (+ 2862us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:15.665621 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:15.665638 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:02:15.665853 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.665859 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.667219 (+ 1360us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.667223 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:15.668048 (+ 825us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.668058 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.668177 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.668192 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 08:02:15.668239 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 08:02:15.668290 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4918,"thread_start_us":92,"threads_started":1}
I20260429 08:02:15.666150 25042 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.654175 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:43985 (local address 127.0.0.1:54434)
0429 08:02:15.659008 (+ 4833us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:15.659050 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:15.659234 (+ 184us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:15.662012 (+ 2778us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:15.662015 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:15.662028 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:02:15.662276 (+ 248us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.662282 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.663664 (+ 1382us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.663669 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:15.664578 (+ 909us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:15.664588 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.665864 (+ 1276us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.665885 (+ 21us) client_negotiation.cc:770] Sending connection context
0429 08:02:15.665962 (+ 77us) client_negotiation.cc:241] Negotiation successful
0429 08:02:15.666019 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4668,"thread_start_us":103,"threads_started":1}
I20260429 08:02:15.670814 24962 tablet_service.cc:1511] Processing CreateTablet for tablet b512d8690c32482182dfe077e25124c9 (DEFAULT_TABLE table=test-table [id=0bd2db116ba645f2b20f9e7ff679f48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:02:15.671869 24962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b512d8690c32482182dfe077e25124c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:02:15.671960 24680 tablet_service.cc:1511] Processing CreateTablet for tablet b512d8690c32482182dfe077e25124c9 (DEFAULT_TABLE table=test-table [id=0bd2db116ba645f2b20f9e7ff679f48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:02:15.672914 24680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b512d8690c32482182dfe077e25124c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:02:15.680688 25046 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:15.658195 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.12.12.130:37159)
0429 08:02:15.665016 (+ 6821us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:15.665024 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:15.665047 (+ 23us) server_negotiation.cc:408] Connection header received
0429 08:02:15.665135 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:15.665140 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:15.665321 (+ 181us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:15.665485 (+ 164us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:02:15.666016 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.667081 (+ 1065us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:15.668186 (+ 1105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:15.680189 (+ 12003us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:15.680317 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:15.680406 (+ 89us) server_negotiation.cc:300] Negotiation successful
0429 08:02:15.680510 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6708,"thread_start_us":68,"threads_started":1}
I20260429 08:02:15.683955 24823 tablet_service.cc:1511] Processing CreateTablet for tablet b512d8690c32482182dfe077e25124c9 (DEFAULT_TABLE table=test-table [id=0bd2db116ba645f2b20f9e7ff679f48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 08:02:15.685187 25048 tablet_bootstrap.cc:492] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: Bootstrap starting.
I20260429 08:02:15.686393 24823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b512d8690c32482182dfe077e25124c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:02:15.689618 25047 tablet_bootstrap.cc:492] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Bootstrap starting.
I20260429 08:02:15.692237 25047 tablet_bootstrap.cc:654] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Neither blocks nor log segments found. Creating new log.
I20260429 08:02:15.694622 25047 log.cc:826] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Log is configured to *not* fsync() on all Append() calls
I20260429 08:02:15.696019 25049 tablet_bootstrap.cc:492] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: Bootstrap starting.
I20260429 08:02:15.699396 25049 tablet_bootstrap.cc:654] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: Neither blocks nor log segments found. Creating new log.
I20260429 08:02:15.700470 25049 log.cc:826] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: Log is configured to *not* fsync() on all Append() calls
I20260429 08:02:15.702955 25048 tablet_bootstrap.cc:654] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: Neither blocks nor log segments found. Creating new log.
I20260429 08:02:15.703817 25048 log.cc:826] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: Log is configured to *not* fsync() on all Append() calls
I20260429 08:02:15.709497 25047 tablet_bootstrap.cc:492] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: No bootstrap required, opened a new log
I20260429 08:02:15.709553 25049 tablet_bootstrap.cc:492] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: No bootstrap required, opened a new log
I20260429 08:02:15.709738 25049 ts_tablet_manager.cc:1403] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260429 08:02:15.709739 25047 ts_tablet_manager.cc:1403] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260429 08:02:15.713104 25048 tablet_bootstrap.cc:492] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: No bootstrap required, opened a new log
I20260429 08:02:15.713346 25047 raft_consensus.cc:359] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:15.713584 25047 raft_consensus.cc:385] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:02:15.713636 25047 raft_consensus.cc:740] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023fdfed641f48688881844d1ceb1534, State: Initialized, Role: FOLLOWER
I20260429 08:02:15.714352 25048 ts_tablet_manager.cc:1403] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20260429 08:02:15.714286 25047 consensus_queue.cc:260] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:15.715314 25047 ts_tablet_manager.cc:1434] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260429 08:02:15.715377 25049 raft_consensus.cc:359] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:15.715605 25049 raft_consensus.cc:385] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:02:15.715682 25049 raft_consensus.cc:740] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1bff319c33451e9611edc45f06bdd8, State: Initialized, Role: FOLLOWER
I20260429 08:02:15.715924 25028 heartbeater.cc:499] Master 127.12.12.190:41057 was elected leader, sending a full tablet report...
I20260429 08:02:15.716197 25049 consensus_queue.cc:260] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:15.717068 24890 heartbeater.cc:499] Master 127.12.12.190:41057 was elected leader, sending a full tablet report...
I20260429 08:02:15.717180 25049 ts_tablet_manager.cc:1434] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260429 08:02:15.718899 25048 raft_consensus.cc:359] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:15.719125 25048 raft_consensus.cc:385] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:02:15.719192 25048 raft_consensus.cc:740] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda8038c9a174145b0a2e6272d21719f, State: Initialized, Role: FOLLOWER
I20260429 08:02:15.733304 25048 consensus_queue.cc:260] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:15.734691 25048 ts_tablet_manager.cc:1434] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260429 08:02:15.735184 24752 heartbeater.cc:499] Master 127.12.12.190:41057 was elected leader, sending a full tablet report...
W20260429 08:02:15.757208 24753 tablet.cc:2404] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 08:02:15.809508 25029 tablet.cc:2404] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 08:02:15.896754 24891 tablet.cc:2404] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 08:02:16.001693 25053 raft_consensus.cc:493] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 08:02:16.001925 25053 raft_consensus.cc:515] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:16.003348 25053 leader_election.cc:290] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bda8038c9a174145b0a2e6272d21719f (127.12.12.129:43985), 0f1bff319c33451e9611edc45f06bdd8 (127.12.12.130:37159)
I20260429 08:02:16.007282 25030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.003829 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:43985 (local address 127.12.12.131:58193)
0429 08:02:16.003957 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:16.003976 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:16.004084 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:16.004641 (+ 557us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:16.004645 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:16.004665 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:02:16.004969 (+ 304us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:16.004977 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.006000 (+ 1023us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:16.006003 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:16.006895 (+ 892us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:16.006905 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.007013 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:16.007032 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 08:02:16.007081 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 08:02:16.007135 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260429 08:02:16.009094 25043 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.004178 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:58193 (local address 127.12.12.129:43985)
0429 08:02:16.004310 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:16.004315 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:16.004333 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:02:16.004387 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:16.004390 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:16.004439 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:16.004525 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:02:16.005112 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.005872 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:16.008299 (+ 2427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.008791 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:16.008853 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:16.008907 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 08:02:16.008969 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 08:02:16.009688 24706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b512d8690c32482182dfe077e25124c9" candidate_uuid: "023fdfed641f48688881844d1ceb1534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda8038c9a174145b0a2e6272d21719f" is_pre_election: true
I20260429 08:02:16.009979 24706 raft_consensus.cc:2468] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023fdfed641f48688881844d1ceb1534 in term 0.
I20260429 08:02:16.010661 24916 leader_election.cc:304] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023fdfed641f48688881844d1ceb1534, bda8038c9a174145b0a2e6272d21719f; no voters:
I20260429 08:02:16.011008 25053 raft_consensus.cc:2804] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 08:02:16.011085 25053 raft_consensus.cc:493] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 08:02:16.011127 25053 raft_consensus.cc:3060] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:02:16.012005 25030 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.004254 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:37159 (local address 127.12.12.131:45749)
0429 08:02:16.007456 (+ 3202us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:16.007470 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 08:02:16.007548 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:02:16.009459 (+ 1911us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:02:16.009462 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:02:16.009482 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 08:02:16.009698 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:16.009706 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.010791 (+ 1085us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:02:16.010795 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:02:16.011640 (+ 845us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:02:16.011650 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.011759 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:16.011775 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 08:02:16.011824 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 08:02:16.011875 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3144,"thread_start_us":89,"threads_started":1}
I20260429 08:02:16.012420 25053 raft_consensus.cc:515] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:16.013012 25046 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.004601 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.131:45749 (local address 127.12.12.130:37159)
0429 08:02:16.004714 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:16.004718 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:16.007721 (+ 3003us) server_negotiation.cc:408] Connection header received
0429 08:02:16.007783 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:16.007786 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:16.007840 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:16.007907 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 08:02:16.009848 (+ 1941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.010612 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:16.012224 (+ 1612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.012724 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:16.012760 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:16.012808 (+ 48us) server_negotiation.cc:300] Negotiation successful
0429 08:02:16.012883 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260429 08:02:16.013650 24828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b512d8690c32482182dfe077e25124c9" candidate_uuid: "023fdfed641f48688881844d1ceb1534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1bff319c33451e9611edc45f06bdd8" is_pre_election: true
I20260429 08:02:16.013918 24828 raft_consensus.cc:2468] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023fdfed641f48688881844d1ceb1534 in term 0.
I20260429 08:02:16.014220 24706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b512d8690c32482182dfe077e25124c9" candidate_uuid: "023fdfed641f48688881844d1ceb1534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda8038c9a174145b0a2e6272d21719f"
I20260429 08:02:16.014269 24841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b512d8690c32482182dfe077e25124c9" candidate_uuid: "023fdfed641f48688881844d1ceb1534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1bff319c33451e9611edc45f06bdd8"
I20260429 08:02:16.014341 24706 raft_consensus.cc:3060] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:02:16.014370 24841 raft_consensus.cc:3060] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:02:16.015501 24706 raft_consensus.cc:2468] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023fdfed641f48688881844d1ceb1534 in term 1.
I20260429 08:02:16.015501 24841 raft_consensus.cc:2468] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023fdfed641f48688881844d1ceb1534 in term 1.
I20260429 08:02:16.016170 24918 leader_election.cc:304] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023fdfed641f48688881844d1ceb1534, 0f1bff319c33451e9611edc45f06bdd8; no voters:
I20260429 08:02:16.016726 25053 leader_election.cc:290] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [CANDIDATE]: Term 1 election: Requested vote from peers bda8038c9a174145b0a2e6272d21719f (127.12.12.129:43985), 0f1bff319c33451e9611edc45f06bdd8 (127.12.12.130:37159)
I20260429 08:02:16.016916 25053 raft_consensus.cc:2804] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:02:16.017006 25053 raft_consensus.cc:697] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 1 LEADER]: Becoming Leader. State: Replica: 023fdfed641f48688881844d1ceb1534, State: Running, Role: LEADER
I20260429 08:02:16.017310 25053 consensus_queue.cc:237] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } }
I20260429 08:02:16.020615 24522 catalog_manager.cc:5671] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 reported cstate change: term changed from 0 to 1, leader changed from <none> to 023fdfed641f48688881844d1ceb1534 (127.12.12.131). New cstate: current_term: 1 leader_uuid: "023fdfed641f48688881844d1ceb1534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023fdfed641f48688881844d1ceb1534" member_type: VOTER last_known_addr { host: "127.12.12.131" port: 34587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 } health_report { overall_health: UNKNOWN } } }
I20260429 08:02:16.044374 25044 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.039884 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.12.12.131:34587)
0429 08:02:16.040028 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 08:02:16.040033 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 08:02:16.040051 (+ 18us) server_negotiation.cc:408] Connection header received
0429 08:02:16.040140 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:02:16.040145 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:02:16.040212 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:02:16.040317 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 08:02:16.040953 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.041707 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:02:16.042789 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:02:16.043036 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:02:16.043139 (+ 103us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 08:02:16.043625 (+ 486us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 08:02:16.043734 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 08:02:16.044155 (+ 421us) server_negotiation.cc:300] Negotiation successful
0429 08:02:16.044207 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 08:02:16.058920 24841 raft_consensus.cc:1275] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 1 FOLLOWER]: Refusing update from remote peer 023fdfed641f48688881844d1ceb1534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:02:16.059831 24706 raft_consensus.cc:1275] T b512d8690c32482182dfe077e25124c9 P bda8038c9a174145b0a2e6272d21719f [term 1 FOLLOWER]: Refusing update from remote peer 023fdfed641f48688881844d1ceb1534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 08:02:16.060653 25053 consensus_queue.cc:1048] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda8038c9a174145b0a2e6272d21719f" member_type: VOTER last_known_addr { host: "127.12.12.129" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:02:16.061681 25053 consensus_queue.cc:1048] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f1bff319c33451e9611edc45f06bdd8" member_type: VOTER last_known_addr { host: "127.12.12.130" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 08:02:16.082275 25061 mvcc.cc:204] Tried to move back new op lower bound from 7280434118859677696 to 7280434118729801728. Current Snapshot: MvccSnapshot[applied={T|T < 7280434118859677696}]
I20260429 08:02:16.084040 25064 mvcc.cc:204] Tried to move back new op lower bound from 7280434118859677696 to 7280434118729801728. Current Snapshot: MvccSnapshot[applied={T|T < 7280434118859677696}]
I20260429 08:02:16.100198 24522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:02:16.100368 24522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 08:02:16.103281 24522 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1e01185d165e436db285add8f325d7d9: Sending DeleteTablet for 3 replicas of tablet b512d8690c32482182dfe077e25124c9
I20260429 08:02:16.104526 24962 tablet_service.cc:1558] Processing DeleteTablet for tablet b512d8690c32482182dfe077e25124c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:02:16 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:53396
I20260429 08:02:16.105047 24680 tablet_service.cc:1558] Processing DeleteTablet for tablet b512d8690c32482182dfe077e25124c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:02:16 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:54434
I20260429 08:02:16.105517 24823 tablet_service.cc:1558] Processing DeleteTablet for tablet b512d8690c32482182dfe077e25124c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 08:02:16 UTC) from {username='kudu', principal='kudu/127.12.12.190@KRBTEST.COM'} at 127.0.0.1:36912
I20260429 08:02:16.105618 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 24618
W20260429 08:02:16.116987 24508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.12.129:43985 (error 108)
I20260429 08:02:16.125101 25072 tablet_replica.cc:333] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8: stopping tablet replica
I20260429 08:02:16.125492 25070 tablet_replica.cc:333] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: stopping tablet replica
I20260429 08:02:16.125922 25072 raft_consensus.cc:2243] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 08:02:16.126267 25070 raft_consensus.cc:2243] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 1 LEADER]: Raft consensus shutting down.
I20260429 08:02:16.126731 25072 raft_consensus.cc:2272] T b512d8690c32482182dfe077e25124c9 P 0f1bff319c33451e9611edc45f06bdd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:02:16.127177 25070 raft_consensus.cc:2272] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 08:02:16.129051 25070 ts_tablet_manager.cc:1916] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 08:02:16.131976 25070 ts_tablet_manager.cc:1929] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 08:02:16.132189 25070 log.cc:1199] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Deleting WAL directory at /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777449587217345-12338-0/minicluster-data/ts-2/wal/wals/b512d8690c32482182dfe077e25124c9
I20260429 08:02:16.132270 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 24760
I20260429 08:02:16.135216 25070 ts_tablet_manager.cc:1950] T b512d8690c32482182dfe077e25124c9 P 023fdfed641f48688881844d1ceb1534: Deleting consensus metadata
I20260429 08:02:16.136669 24507 catalog_manager.cc:5002] TS 023fdfed641f48688881844d1ceb1534 (127.12.12.131:34587): tablet b512d8690c32482182dfe077e25124c9 (table test-table [id=0bd2db116ba645f2b20f9e7ff679f48f]) successfully deleted
I20260429 08:02:16.138561 25041 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.137548 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.129:43985 (local address 127.0.0.1:54450)
0429 08:02:16.138398 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:16.138469 (+ 71us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.129:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":753}
W20260429 08:02:16.138903 24508 catalog_manager.cc:4729] TS bda8038c9a174145b0a2e6272d21719f (127.12.12.129:43985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b512d8690c32482182dfe077e25124c9: Network error: Client connection negotiation failed: client connection to 127.12.12.129:43985: connect: Connection refused (error 111)
I20260429 08:02:16.149925 25041 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:02:16.149633 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.130:37159 (local address 127.0.0.1:36914)
0429 08:02:16.149780 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0429 08:02:16.149827 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.130:37159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260429 08:02:16.153033 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 24897
I20260429 08:02:16.162317 12338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskPttFKW/build/debug/bin/kudu with pid 24490
I20260429 08:02:16.170588 12338 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260429 08:02:16.546838 12338 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260429 08:02:16.546978 12338 mini_ranger.cc:66] Stopping Ranger...
I20260429 08:02:16.893923 12338 mini_ranger.cc:68] Stopped Ranger
2026-04-29 08:02:16.894 UTC [23800] LOG: received smart shutdown request
2026-04-29 08:02:16.897 UTC [23800] LOG: background worker "logical replication launcher" (PID 23808) exited with exit code 1
2026-04-29 08:02:16.898 UTC [23803] LOG: shutting down
2026-04-29 08:02:16.900 UTC [23803] LOG: checkpoint starting: shutdown immediate
2026-04-29 08:02:17.866 UTC [23803] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.042 s, sync=0.919 s, total=0.969 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12073 kB, estimate=12073 kB; lsn=0/20450C8, redo lsn=0/20450C8
2026-04-29 08:02:17.875 UTC [23800] LOG: database system is shut down
2026-04-29T08:02:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54537 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 08:02:18 dist-test-slave-nhqz krb5kdc[25082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 08:02:18 dist-test-slave-nhqz krb5kdc[25082](info): set up 2 sockets
Apr 29 08:02:18 dist-test-slave-nhqz krb5kdc[25082](info): commencing operation
krb5kdc: starting...
W20260429 08:02:19.985755 12338 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/test-user.keytab.
Apr 29 08:02:20 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T08:02:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T08:02:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/rangeradmin_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/rangeradmin_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/rangerlookup_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/rangerlookup_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
I20260429 08:02:20.147667 12338 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPttFKW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/postgres -l logfile start
2026-04-29 08:02:22.683 UTC [25113] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-29 08:02:22.684 UTC [25113] LOG: listening on IPv4 address "127.12.12.148", port 41731
2026-04-29 08:02:22.687 UTC [25113] LOG: listening on Unix socket "/tmp/.s.PGSQL.41731"
2026-04-29 08:02:22.692 UTC [25118] LOG: database system was shut down at 2026-04-29 08:02:21 UTC
2026-04-29 08:02:22.696 UTC [25113] LOG: database system is ready to accept connections
I20260429 08:02:24.585425 12338 mini_postgres.cc:96] Postgres bound to 41731
2026-04-29 08:02:24.591 UTC [25124] FATAL: database "slave" does not exist
127.12.12.148:41731 - accepting connections
I20260429 08:02:24.591878 12338 mini_ranger.cc:161] Starting Ranger...
I20260429 08:02:24.609496 12338 mini_ranger.cc:84] Created miniranger Postgres user
I20260429 08:02:24.679644 12338 mini_ranger.cc:87] Created ranger Postgres database
I20260429 08:02:24.679741 12338 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-admin
2026-04-29 08:02:24,992 [I] DB FLAVOR :POSTGRES
2026-04-29 08:02:24,993 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 08:02:24,993 [I] Checking connection..
2026-04-29 08:02:24,993 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-29 08:02:25,318 [I] Checking connection passed.
2026-04-29 08:02:25,318 [I] --------- Verifying version history table ---------
2026-04-29 08:02:25,318 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 08:02:25,636 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 08:02:25,636 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 08:02:25,955 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 08:02:25,955 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-29 08:02:25,955 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-29 08:02:26.254 UTC [25237] WARNING: there is no transaction in progress
2026-04-29 08:02:26,279 [I] create_dbversion_catalog.sql file imported successfully
2026-04-29 08:02:26,280 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 08:02:26,623 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-29 08:02:26,623 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 08:02:26,624 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-29 08:02:26,907 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-29 08:02:27,229 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nhqz.c.gcp-upstream.internal','N') ;"
2026-04-29 08:02:27,507 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 08:02:27,853 [I] Table x_portal_user does not exist in database ranger
2026-04-29 08:02:27,854 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 08:02:28,204 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-29 08:02:28,204 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 08:02:28,521 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 08:02:28,851 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-29 08:02:28,852 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-29 08:02:29.742 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.753 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.765 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.775 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.787 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.798 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.843 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.849 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.858 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.868 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.876 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.886 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.893 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.898 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.901 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:29.905 UTC [25436] WARNING: there is no transaction in progress
2026-04-29 08:02:30,322 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-29 08:02:30,322 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-29 08:02:30,649 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-29 08:02:30,649 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 08:02:30,963 [I] Table x_portal_user already exists in database 'ranger'
2026-04-29 08:02:30,963 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 08:02:31,261 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-29 08:02:31,261 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 08:02:31,569 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 08:02:31,926 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nhqz.c.gcp-upstream.internal';"
2026-04-29 08:02:32,249 [I] CORE_DB_SCHEMA import status has been updated
2026-04-29 08:02:32,249 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-29 08:02:32,601 [I] DB_PATCHES have already been applied
I20260429 08:02:32.606683 12338 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskPttFKW/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260429 08:02:32.606753 12338 mini_ranger.cc:193] Using host: 127.12.12.148
I20260429 08:02:32.609579 12338 mini_ranger.cc:239] Ranger admin URL: http://127.12.12.148:59545
Apr 29, 2026 8:02:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 8:02:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 8:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp
Apr 29, 2026 8:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 29, 2026 8:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 29, 2026 8:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-admin/ews/webapp].
Apr 29, 2026 8:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260429 08:02:34.924466 12338 mini_ranger.cc:160] Time spent starting Ranger: real 10.333s user 0.000s sys 0.010s
Apr 29, 2026 8:02:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 29, 2026 8:02:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Apr 29, 2026 8:02:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 29, 2026 8:02:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 29, 2026 8:02:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260429 08:02:59.537292 12338 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/rangerkms_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/rangerkms_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.12.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.12.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.12.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/HTTP_127.12.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Apr 29 08:02:59 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449779, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260429 08:02:59.639561 12338 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260429 08:02:59.651496 12338 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260429 08:02:59.779023 12338 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260429 08:02:59.779102 12338 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms
I20260429 08:02:59.779119 12338 mini_ranger_kms.cc:227] Using postgres at 127.12.12.148:41731
2026-04-29 08:03:00,001 [I] DB FLAVOR :POSTGRES
2026-04-29 08:03:00,002 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 08:03:00,002 [I] Checking connection
2026-04-29 08:03:00,002 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-04-29 08:03:00,462 [I] connection success
2026-04-29 08:03:00,462 [I] --------- Verifying Ranger DB tables ---------
2026-04-29 08:03:00,462 [I] Verifying table ranger_masterkey in database rangerkms
2026-04-29 08:03:00,462 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-04-29 08:03:00,849 [I] Table ranger_masterkey does not exist in database rangerkms
2026-04-29 08:03:00,849 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 08:03:00,849 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-04-29 08:03:00,849 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.12.148:41731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-04-29 08:03:01,192 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260429 08:03:01.699271 12338 mini_ranger_kms.cc:326] Created kms service
I20260429 08:03:01.923736 12338 mini_ranger_kms.cc:342] Created kudu user
I20260429 08:03:01.969681 12338 mini_ranger_kms.cc:359] Created rangerkms user
Apr 29 08:03:01 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449779, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:03:02.441886 12338 mini_ranger_kms.cc:400] Added ranger policy
I20260429 08:03:02.442124 12338 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPttFKW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPttFKW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260429 08:03:02.442162 12338 mini_ranger_kms.cc:240] Using host: 127.12.12.148
I20260429 08:03:02.446303 12338 mini_ranger_kms.cc:292] Ranger KMS PID: 25738
I20260429 08:03:02.446401 12338 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.12.12.148:50461
Apr 29, 2026 8:03:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 8:03:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 8:03:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Apr 29, 2026 8:03:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/ews/webapp] .....
Apr 29, 2026 8:03:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-kms/ews/webapp].
Apr 29, 2026 8:03:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260429 08:03:04.721644 12338 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.082s user 0.019s sys 0.174s
I20260429 08:03:04.721764 12338 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260429 08:03:04.721833 12338 mini_ranger_kms.cc:417] 127.12.12.148:50461/kms/v1/keys
Apr 29 08:03:05 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Apr 29 08:03:05 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.148@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
Apr 29 08:03:06 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777449785, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.12.148@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
Apr 29 08:03:09 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:03:09.127516 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:33353
--webserver_interface=127.12.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.12.190:33353
--ranger_config_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:50461/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:03:09.232270 25805 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:03:09.232507 25805 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:03:09.232558 25805 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:03:09.235850 25805 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 08:03:09.235913 25805 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:03:09.235935 25805 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:03:09.235953 25805 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 08:03:09.236006 25805 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 08:03:09.240629 25805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.12.148:50461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.12.190:33353
--ranger_config_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.12.190:33353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.12.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:03:09.241752 25805 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:03:09.242657 25805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:03:09.248493 25813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:03:09.248548 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:03:09.248718 25805 server_base.cc:1061] running on GCE node
W20260429 08:03:09.248566 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:03:09.249289 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:03:09.250147 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:03:09.251344 25805 hybrid_clock.cc:648] HybridClock initialized: now 1777449789251339 us; error 29 us; skew 500 ppm
Apr 29 08:03:09 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449789, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:03:09.253955 25805 init.cc:377] Logged in from keytab as kudu/127.12.12.190@KRBTEST.COM (short username kudu)
I20260429 08:03:09.254926 25805 webserver.cc:492] Webserver started at http://127.12.12.190:37349/ using document root <none> and password file <none>
I20260429 08:03:09.255463 25805 fs_manager.cc:362] Metadata directory not provided
I20260429 08:03:09.255537 25805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:03:09.255735 25805 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:03:09 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449789, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.190@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:03:09.305352 25805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data/instance:
uuid: "f2abec8c99c74c5494fc448575cf6d38"
format_stamp: "Formatted at 2026-04-29 08:03:09 on dist-test-slave-nhqz"
server_key: "60b6e2ac5c14f96ec0e17ec8f506615a"
server_key_iv: "51afa6dff822dbc9845fc804493929e0"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:09.305903 25805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal/instance:
uuid: "f2abec8c99c74c5494fc448575cf6d38"
format_stamp: "Formatted at 2026-04-29 08:03:09 on dist-test-slave-nhqz"
server_key: "60b6e2ac5c14f96ec0e17ec8f506615a"
server_key_iv: "51afa6dff822dbc9845fc804493929e0"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:09.310056 25805 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260429 08:03:09.327203 25821 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:03:09.328464 25805 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260429 08:03:09.328586 25805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal
uuid: "f2abec8c99c74c5494fc448575cf6d38"
format_stamp: "Formatted at 2026-04-29 08:03:09 on dist-test-slave-nhqz"
server_key: "60b6e2ac5c14f96ec0e17ec8f506615a"
server_key_iv: "51afa6dff822dbc9845fc804493929e0"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:09.328683 25805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:03:09.344281 25805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:03:09.350389 25805 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:03:09.350559 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:03:09.358147 25873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.190:33353 every 8 connection(s)
I20260429 08:03:09.358155 25805 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.190:33353
I20260429 08:03:09.359166 25805 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/data/info.pb
I20260429 08:03:09.361869 25874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 08:03:09.365149 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 25805
I20260429 08:03:09.365288 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/wal/instance
I20260429 08:03:09.365554 12338 external_mini_cluster.cc:1468] Setting key 4a9cc886763ed344eacb54e2df2c4b70
I20260429 08:03:09.367235 25874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38: Bootstrap starting.
I20260429 08:03:09.369745 25874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38: Neither blocks nor log segments found. Creating new log.
I20260429 08:03:09.370477 25874 log.cc:826] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38: Log is configured to *not* fsync() on all Append() calls
I20260429 08:03:09.372567 25874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38: No bootstrap required, opened a new log
Apr 29 08:03:09 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:03:09.375595 25874 raft_consensus.cc:359] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2abec8c99c74c5494fc448575cf6d38" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33353 } }
I20260429 08:03:09.375770 25874 raft_consensus.cc:385] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 08:03:09.375854 25874 raft_consensus.cc:740] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2abec8c99c74c5494fc448575cf6d38, State: Initialized, Role: FOLLOWER
I20260429 08:03:09.376339 25874 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2abec8c99c74c5494fc448575cf6d38" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33353 } }
I20260429 08:03:09.376499 25874 raft_consensus.cc:399] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 08:03:09.376564 25874 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 08:03:09.376708 25874 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 0 FOLLOWER]: Advancing to term 1
I20260429 08:03:09.377703 25874 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2abec8c99c74c5494fc448575cf6d38" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33353 } }
I20260429 08:03:09.378067 25874 leader_election.cc:304] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2abec8c99c74c5494fc448575cf6d38; no voters:
I20260429 08:03:09.378377 25874 leader_election.cc:290] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 08:03:09.378455 25879 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 08:03:09.378643 25879 raft_consensus.cc:697] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [term 1 LEADER]: Becoming Leader. State: Replica: f2abec8c99c74c5494fc448575cf6d38, State: Running, Role: LEADER
I20260429 08:03:09.378790 25877 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:09.366807 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.12.12.190:33353)
0429 08:03:09.367255 (+ 448us) server_negotiation.cc:207] Beginning negotiation
0429 08:03:09.367264 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 08:03:09.367305 (+ 41us) server_negotiation.cc:408] Connection header received
0429 08:03:09.367897 (+ 592us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:03:09.367914 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:03:09.368189 (+ 275us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:03:09.368434 (+ 245us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:03:09.369547 (+ 1113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:03:09.370343 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:03:09.371340 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:03:09.371614 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:03:09.373862 (+ 2248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:03:09.373880 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:03:09.373890 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:03:09.373917 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:03:09.376210 (+ 2293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:03:09.376673 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:03:09.376678 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:03:09.376684 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:03:09.376753 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:03:09.377077 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:03:09.377080 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:03:09.377081 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:03:09.377359 (+ 278us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:03:09.377514 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0429 08:03:09.377821 (+ 307us) server_negotiation.cc:300] Negotiation successful
0429 08:03:09.378065 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":176,"threads_started":1}
I20260429 08:03:09.378959 25879 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2abec8c99c74c5494fc448575cf6d38" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33353 } }
I20260429 08:03:09.379498 25874 sys_catalog.cc:565] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 08:03:09.380033 25879 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2abec8c99c74c5494fc448575cf6d38. Latest consensus state: current_term: 1 leader_uuid: "f2abec8c99c74c5494fc448575cf6d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2abec8c99c74c5494fc448575cf6d38" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33353 } } }
I20260429 08:03:09.380771 25879 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [sys.catalog]: This master's current role is: LEADER
I20260429 08:03:09.380666 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2abec8c99c74c5494fc448575cf6d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2abec8c99c74c5494fc448575cf6d38" member_type: VOTER last_known_addr { host: "127.12.12.190" port: 33353 } } }
I20260429 08:03:09.381062 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38 [sys.catalog]: This master's current role is: LEADER
I20260429 08:03:09.381850 25888 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 08:03:09.382066 25874 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260429 08:03:09.387344 25888 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 08:03:09.392954 25888 catalog_manager.cc:1357] Generated new cluster ID: 6c70e7828fd2472d9d4d396defc92239
I20260429 08:03:09.393028 25888 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 08:03:09.410245 25888 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 08:03:09.411087 25888 catalog_manager.cc:1514] Loading token signing keys...
I20260429 08:03:09.418427 25888 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f2abec8c99c74c5494fc448575cf6d38: Generated new TSK 0
I20260429 08:03:09.419054 25888 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:03:10.151916 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:50461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.129:0
--local_ip_for_outbound_sockets=127.12.12.129
--webserver_interface=127.12.12.129
--webserver_port=0
--tserver_master_addrs=127.12.12.190:33353
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 08:03:10.302533 25933 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:03:10.302865 25933 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:03:10.302958 25933 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:03:10.307901 25933 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:03:10.307994 25933 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:03:10.308120 25933 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.129
I20260429 08:03:10.314677 25933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.12.148:50461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.12.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:33353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.129
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:03:10.316021 25933 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:03:10.317155 25933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:03:10.324817 25942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:03:10.324877 25940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:03:10.325129 25939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:03:10.326226 25933 server_base.cc:1061] running on GCE node
I20260429 08:03:10.326675 25933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:03:10.327354 25933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:03:10.328656 25933 hybrid_clock.cc:648] HybridClock initialized: now 1777449790328597 us; error 73 us; skew 500 ppm
Apr 29 08:03:10 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449790, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:03:10.332152 25933 init.cc:377] Logged in from keytab as kudu/127.12.12.129@KRBTEST.COM (short username kudu)
I20260429 08:03:10.333657 25933 webserver.cc:492] Webserver started at http://127.12.12.129:45027/ using document root <none> and password file <none>
I20260429 08:03:10.334442 25933 fs_manager.cc:362] Metadata directory not provided
I20260429 08:03:10.334568 25933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:03:10.334862 25933 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:03:10 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449790, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:03:10.356629 25933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data/instance:
uuid: "7d50b8c61054463aa66d89c94780c8b3"
format_stamp: "Formatted at 2026-04-29 08:03:10 on dist-test-slave-nhqz"
server_key: "bcf82e25d8eb7ae9bc4b8c4a82d643fd"
server_key_iv: "7f2750b0b1b3832a2f452f76119c187b"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:10.357535 25933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance:
uuid: "7d50b8c61054463aa66d89c94780c8b3"
format_stamp: "Formatted at 2026-04-29 08:03:10 on dist-test-slave-nhqz"
server_key: "bcf82e25d8eb7ae9bc4b8c4a82d643fd"
server_key_iv: "7f2750b0b1b3832a2f452f76119c187b"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:10.382522 25933 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260429 08:03:10.406306 25950 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.001s user 0.000s sys 0.000s
I20260429 08:03:10.418215 25933 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.002s sys 0.000s
I20260429 08:03:10.418396 25933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal
uuid: "7d50b8c61054463aa66d89c94780c8b3"
format_stamp: "Formatted at 2026-04-29 08:03:10 on dist-test-slave-nhqz"
server_key: "bcf82e25d8eb7ae9bc4b8c4a82d643fd"
server_key_iv: "7f2750b0b1b3832a2f452f76119c187b"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:10.418521 25933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:03:10.470260 25933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:03:10.473923 25933 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:03:10.475148 25933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:03:10.475903 25933 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:03:10.477123 25933 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:03:10.477238 25933 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:03:10.477385 25933 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:03:10.477478 25933 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:03:10.489092 25933 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.129:42739
I20260429 08:03:10.490485 25933 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/data/info.pb
I20260429 08:03:10.491412 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 25933
I20260429 08:03:10.491550 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-0/wal/instance
I20260429 08:03:10.491791 12338 external_mini_cluster.cc:1468] Setting key 96d2040ff2c150c39661a660a8fc69d7
I20260429 08:03:10.507112 26066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.129:42739 every 8 connection(s)
WARNING: no policy specified for kudu/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.12.130@KRBTEST.COM" created.
Apr 29 08:03:10 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449790, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.129@KRBTEST.COM for kudu/127.12.12.190@KRBTEST.COM
I20260429 08:03:10.531225 26070 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:10.510307 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.129:58129)
0429 08:03:10.516012 (+ 5705us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:10.516059 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 08:03:10.517026 (+ 967us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:03:10.517822 (+ 796us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 08:03:10.517834 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 08:03:10.520297 (+ 2463us) client_negotiation.cc:190] Negotiated authn=SASL
0429 08:03:10.521040 (+ 743us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:03:10.521059 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:03:10.522445 (+ 1386us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 08:03:10.522449 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 08:03:10.523089 (+ 640us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 08:03:10.523097 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 08:03:10.523326 (+ 229us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:03:10.524390 (+ 1064us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 08:03:10.524415 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0429 08:03:10.525975 (+ 1560us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 08:03:10.528267 (+ 2292us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:03:10.528275 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:03:10.528297 (+ 22us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:03:10.528644 (+ 347us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:03:10.528975 (+ 331us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 08:03:10.528979 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 08:03:10.528982 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 08:03:10.529119 (+ 137us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 08:03:10.529564 (+ 445us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 08:03:10.529570 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 08:03:10.529856 (+ 286us) client_negotiation.cc:770] Sending connection context
0429 08:03:10.530086 (+ 230us) client_negotiation.cc:241] Negotiation successful
0429 08:03:10.530317 (+ 231us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5481,"thread_start_us":154,"threads_started":1}
I20260429 08:03:10.532217 26071 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:10.511434 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.12.12.129:58129 (local address 127.12.12.190:33353)
0429 08:03:10.511731 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0429 08:03:10.511737 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 08:03:10.516197 (+ 4460us) server_negotiation.cc:408] Connection header received
0429 08:03:10.517265 (+ 1068us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 08:03:10.517268 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 08:03:10.517409 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 08:03:10.517493 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 08:03:10.521230 (+ 3737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:03:10.522256 (+ 1026us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 08:03:10.523820 (+ 1564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 08:03:10.524024 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 08:03:10.526157 (+ 2133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 08:03:10.526181 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 08:03:10.526185 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 08:03:10.526211 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 08:03:10.528111 (+ 1900us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:03:10.528760 (+ 649us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:03:10.528764 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:03:10.528766 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:03:10.528863 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 08:03:10.529225 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 08:03:10.529228 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 08:03:10.529230 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 08:03:10.529429 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 08:03:10.529511 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 08:03:10.531935 (+ 2424us) server_negotiation.cc:300] Negotiation successful
0429 08:03:10.532058 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260429 08:03:10.534049 26067 heartbeater.cc:344] Connected to a master server at 127.12.12.190:33353
I20260429 08:03:10.534329 26067 heartbeater.cc:461] Registering TS with master...
I20260429 08:03:10.534991 26067 heartbeater.cc:507] Master 127.12.12.190:33353 requested a full tablet report, sending...
I20260429 08:03:10.536834 25838 ts_manager.cc:194] Registered new tserver with Master: 7d50b8c61054463aa66d89c94780c8b3 (127.12.12.129:42739)
I20260429 08:03:10.538457 25838 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.12.129@KRBTEST.COM'} at 127.12.12.129:58129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.12.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.12.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab.
I20260429 08:03:10.583954 12338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskPttFKW/build/debug/bin/kudu
/tmp/dist-test-taskPttFKW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.12.148:50461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.12.130:0
--local_ip_for_outbound_sockets=127.12.12.130
--webserver_interface=127.12.12.130
--webserver_port=0
--tserver_master_addrs=127.12.12.190:33353
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.12.12.148:33473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.12.12.190 from keytab /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:878)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:745)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20260429 08:03:10.594139 25925 server.cc:273] Received an EOF from the subprocess
F20260429 08:03:10.595093 25898 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f3030e49dcd google::LogMessage::Fail() at ??:0
@ 0x7f3030e4db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3030e497cc google::LogMessage::Flush() at ??:0
@ 0x7f3030e4af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3032c63157 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f3032c5ff59 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f3032c63e85 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557fb3a07244 std::function<>::operator()() at ??:0
@ 0x7f303234e7ea kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3032c936db start_thread at ??:0
@ 0x7f303000061f clone at ??:0
W20260429 08:03:10.701263 26075 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 08:03:10.701524 26075 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 08:03:10.701612 26075 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 08:03:10.704959 26075 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 08:03:10.705061 26075 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 08:03:10.705188 26075 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.12.130
I20260429 08:03:10.709609 26075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.12.148:33473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.12.148:50461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/krb5kdc/kudu.keytab
--principal=kudu/127.12.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.12.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.12.12.190:33353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.12.130
--log_dir=/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 80f39681e127423fa66f8c177be6c05697ac70dd
build type DEBUG
built by None at 29 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11656
I20260429 08:03:10.710736 26075 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 08:03:10.711599 26075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 08:03:10.717976 26093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:03:10.717985 26092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 08:03:10.717983 26095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 08:03:10.718541 26075 server_base.cc:1061] running on GCE node
I20260429 08:03:10.718943 26075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 08:03:10.719612 26075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 08:03:10.720826 26075 hybrid_clock.cc:648] HybridClock initialized: now 1777449790720782 us; error 33 us; skew 500 ppm
Apr 29 08:03:10 dist-test-slave-nhqz krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449790, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 08:03:10.724256 26075 init.cc:377] Logged in from keytab as kudu/127.12.12.130@KRBTEST.COM (short username kudu)
I20260429 08:03:10.725553 26075 webserver.cc:492] Webserver started at http://127.12.12.130:38365/ using document root <none> and password file <none>
I20260429 08:03:10.726140 26075 fs_manager.cc:362] Metadata directory not provided
I20260429 08:03:10.726215 26075 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 08:03:10.726438 26075 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 08:03:10 dist-test-slave-nhqz krb5kdc[25082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777449790, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.12.130@KRBTEST.COM for HTTP/127.12.12.148@KRBTEST.COM
I20260429 08:03:10.745136 26075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data/instance:
uuid: "b8738101207d4120bc17fe8f7f7b7c17"
format_stamp: "Formatted at 2026-04-29 08:03:10 on dist-test-slave-nhqz"
server_key: "3298ea1ace467081c5e95d8b6fc29648"
server_key_iv: "6c97ea481f945ef7970f94894f0f6926"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:10.745821 26075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance:
uuid: "b8738101207d4120bc17fe8f7f7b7c17"
format_stamp: "Formatted at 2026-04-29 08:03:10 on dist-test-slave-nhqz"
server_key: "3298ea1ace467081c5e95d8b6fc29648"
server_key_iv: "6c97ea481f945ef7970f94894f0f6926"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:10.750468 26075 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260429 08:03:10.765995 26102 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 08:03:10.767359 26075 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260429 08:03:10.767491 26075 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal
uuid: "b8738101207d4120bc17fe8f7f7b7c17"
format_stamp: "Formatted at 2026-04-29 08:03:10 on dist-test-slave-nhqz"
server_key: "3298ea1ace467081c5e95d8b6fc29648"
server_key_iv: "6c97ea481f945ef7970f94894f0f6926"
server_key_version: "kuduclusterkey@0"
I20260429 08:03:10.767617 26075 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 08:03:10.802562 26075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 08:03:10.805930 26075 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 08:03:10.806154 26075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 08:03:10.806783 26075 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 08:03:10.807868 26075 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 08:03:10.807968 26075 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 08:03:10.808063 26075 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 08:03:10.808121 26075 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 08:03:10.819151 26075 rpc_server.cc:307] RPC server started. Bound to: 127.12.12.130:39493
I20260429 08:03:10.819317 26215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.12.130:39493 every 8 connection(s)
I20260429 08:03:10.820173 26075 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/data/info.pb
I20260429 08:03:10.821693 12338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskPttFKW/build/debug/bin/kudu as pid 26075
I20260429 08:03:10.821816 12338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskPttFKW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777449587217345-12338-0/minicluster-data/ts-1/wal/instance
I20260429 08:03:10.822109 12338 external_mini_cluster.cc:1468] Setting key 18b2c030e46c5aabefc377a145e8bc62
W20260429 08:03:10.834973 25080 connection.cc:570] client connection to 127.12.12.190:33353 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260429 08:03:10.836010 26218 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 08:03:10.822991 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:38969)
0429 08:03:10.823429 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:10.823472 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0429 08:03:10.824460 (+ 988us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 08:03:10.835346 (+ 10886us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
W20260429 08:03:10.836478 26216 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:33353 (0 consecutive failures): Network error: Failed to ping master at 127.12.12.190:33353: Client connection negotiation failed: client connection to 127.12.12.190:33353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 08:03:10.837276 26218 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:10.837000 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:60993)
0429 08:03:10.837170 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:10.837220 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260429 08:03:10.837913 26218 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:10.837731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:58427)
0429 08:03:10.837831 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:10.837865 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260429 08:03:10.838094 26216 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 08:03:11.541242 26067 heartbeater.cc:499] Master 127.12.12.190:33353 was elected leader, sending a full tablet report...
I20260429 08:03:11.542363 26220 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:11.541851 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.129:50743)
0429 08:03:11.542171 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:11.542241 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
W20260429 08:03:11.542691 26067 heartbeater.cc:646] Failed to heartbeat to 127.12.12.190:33353 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
I20260429 08:03:11.839160 26221 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:11.838711 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:54487)
0429 08:03:11.838981 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:11.839036 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260429 08:03:12.543928 26222 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:12.543492 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.129:59999)
0429 08:03:12.543778 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:12.543833 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260429 08:03:12.840431 26223 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:12.839932 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:47789)
0429 08:03:12.840280 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:12.840337 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":78,"threads_started":1}
I20260429 08:03:13.545266 26224 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:13.544792 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.129:54035)
0429 08:03:13.545109 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:13.545167 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":76,"threads_started":1}
I20260429 08:03:13.841754 26225 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:13.841337 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:50547)
0429 08:03:13.841599 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:13.841655 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260429 08:03:14.546743 26226 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:14.546252 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.129:51799)
0429 08:03:14.546586 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:14.546645 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260429 08:03:14.843204 26227 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:14.842763 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:59197)
0429 08:03:14.843041 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:14.843102 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260429 08:03:15.548059 26228 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:15.547618 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.129:45183)
0429 08:03:15.547902 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:15.547959 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260429 08:03:15.844547 26229 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 08:03:15.844115 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.12.12.190:33353 (local address 127.12.12.130:46307)
0429 08:03:15.844392 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0429 08:03:15.844448 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.12.190:33353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
F20260429 08:03:15.879724 12338 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1777449795 (unix time) try "date -d @1777449795" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003032) received by PID 12338 (TID 0x7fbc1b92bd80) from PID 12338; stack trace: ***
@ 0x7fbc275be980 (unknown) at ??:0
@ 0x7fbc24c0fe87 gsignal at ??:0
@ 0x7fbc24c117f1 abort at ??:0
@ 0x7fbc25e7cdcd google::LogMessage::Fail() at ??:0
@ 0x7fbc25e80b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fbc25e7c7cc google::LogMessage::Flush() at ??:0
@ 0x7fbc25e7df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbc28c0043b kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7fbc28bfd156 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55deea8be7bf kudu::SecurityITest::StartCluster() at ??:0
@ 0x55deea8a6aa5 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7fbc26e890ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fbc26e7dbdb testing::Test::Run() at ??:0
@ 0x7fbc26e7dd9d testing::TestInfo::Run() at ??:0
@ 0x7fbc26e7e377 testing::TestSuite::Run() at ??:0
@ 0x7fbc26e7e77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fbc26e8960d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fbc26e7de63 testing::UnitTest::Run() at ??:0
@ 0x55deea8f6b0e RUN_ALL_TESTS() at ??:0
@ 0x55deea8f5cfb main at ??:0
@ 0x7fbc24bf2c87 __libc_start_main at ??:0
@ 0x55deea88defa _start at ??:0
2026-04-29 08:03:16.064 UTC [25783] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 08:03:16.065 UTC [25634] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 08:03:16.068 UTC [25633] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 08:03:16.068 UTC [25630] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 08:03:16.069 UTC [25629] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 08:03:16.069 UTC [25631] FATAL: terminating connection due to unexpected postmaster exit